2024-03-09T15:16:25.288 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-09T15:16:25.288 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184 --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --name yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184/orig.config.yaml 2024-03-09T15:16:25.310 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T15:16:25.603 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184 branch: wip-yuri2-testing-2024-03-01-1606 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7587184' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri2-testing-2024-03-01-1606 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a96b1c0427a378792255b04333efc5a5569fd416 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a96b1c0427a378792255b04333efc5a5569fd416 workunit: branch: wip-yuri2-testing-2024-03-01-1606 sha1: a96b1c0427a378792255b04333efc5a5569fd416 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3874 sha1: a96b1c0427a378792255b04333efc5a5569fd416 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-03-01-1606 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a96b1c0427a378792255b04333efc5a5569fd416 targets: smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO829iPB/sWh84BWXixg0pjGsAKBKizpNi5TGVdn+npRBJu41t1RoVagiFjvfgVpmunoj/e8FiMAk3An6Io45Gg= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_16:19:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-09T15:16:25.603 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa; will attempt to use it 2024-03-09T15:16:25.604 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks 2024-03-09T15:16:25.604 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-09T15:16:25.606 INFO:teuthology.task.internal:Checking packages... 2024-03-09T15:16:25.630 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a96b1c0427a378792255b04333efc5a5569fd416' 2024-03-09T15:16:25.630 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-09T15:16:25.630 INFO:teuthology.packaging:ref: None 2024-03-09T15:16:25.630 INFO:teuthology.packaging:tag: None 2024-03-09T15:16:25.630 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-03-01-1606 2024-03-09T15:16:25.630 INFO:teuthology.packaging:sha1: a96b1c0427a378792255b04333efc5a5569fd416 2024-03-09T15:16:25.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri2-testing-2024-03-01-1606 2024-03-09T15:16:25.772 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1817.ga96b1c04 2024-03-09T15:16:25.774 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-09T15:16:25.780 INFO:teuthology.task.internal:no buildpackages task found 2024-03-09T15:16:25.780 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-09T15:16:25.795 INFO:teuthology.task.internal:Saving configuration 2024-03-09T15:16:25.805 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-09T15:16:25.810 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T15:16:25.835 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 15:11:16.655937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO829iPB/sWh84BWXixg0pjGsAKBKizpNi5TGVdn+npRBJu41t1RoVagiFjvfgVpmunoj/e8FiMAk3An6Io45Gg='} 2024-03-09T15:16:25.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-09T15:16:25.841 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-09T15:16:25.841 INFO:teuthology.run_tasks:Running task console_log... 2024-03-09T15:16:25.885 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f76faa5cf70>, signals=[15]) 2024-03-09T15:16:25.886 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-09T15:16:25.891 INFO:teuthology.task.internal:Opening connections... 2024-03-09T15:16:25.891 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-09T15:16:25.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:16:25.984 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-09T15:16:25.990 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-09T15:16:26.054 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-09T15:16:26.055 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-09T15:16:26.112 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-09T15:16:26.112 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-09T15:16:26.113 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-09T15:16:26.113 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-09T15:16:26.113 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-09T15:16:26.113 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-09T15:16:26.113 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-09T15:16:26.114 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-09T15:16:26.114 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-09T15:16:26.114 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-09T15:16:26.114 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-09T15:16:26.114 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-09T15:16:26.115 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-09T15:16:26.115 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-09T15:16:26.115 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-09T15:16:26.139 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-09T15:16:26.147 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-09T15:16:26.152 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-09T15:16:26.152 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-09T15:16:26.169 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-09T15:16:26.177 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-09T15:16:26.178 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-09T15:16:26.304 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-09T15:16:26.337 INFO:teuthology.run_tasks:Running task kernel... 2024-03-09T15:16:26.353 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-09T15:16:26.354 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-09T15:16:26.354 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-09T15:16:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:16:26.370 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-09T15:16:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-09T15:16:26.426 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-03-09T15:16:26.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-03-09T15:16:28.654 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-09T15:16:30.012 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 8.1 MB/s | 2.0 MB 00:00 2024-03-09T15:16:30.936 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-09T15:16:32.613 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-09T15:16:37.221 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.1 MB 00:01 2024-03-09T15:16:38.831 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-09T15:16:40.734 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-09T15:16:45.648 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-09T15:16:47.880 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-09T15:16:47.880 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-09T15:16:47.988 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:16:48.001 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:16:48.001 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-09T15:16:48.001 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:16:48.001 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 100 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 123 M 2024-03-09T15:16:48.002 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:16:49.416 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.8 MB/s | 6.0 MB 00:01 2024-03-09T15:16:52.491 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 4.8 MB/s | 21 MB 00:04 2024-03-09T15:16:53.482 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 8.2 MB/s | 33 MB 00:04 2024-03-09T15:16:55.133 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.6 MB/s | 39 MB 00:06 2024-03-09T15:16:55.133 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:16:55.133 INFO:teuthology.orchestra.run.smithi146.stdout:Total 14 MB/s | 100 MB 00:07 2024-03-09T15:16:56.022 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:16:56.414 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:16:56.414 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:16:58.053 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:16:58.054 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:17:00.818 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:17:01.501 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-09T15:17:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T15:17:04.432 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T15:17:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T15:17:09.370 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T15:17:09.798 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:14.617 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:43.775 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:43.776 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-09T15:17:43.776 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:17:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:44.308 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:44.308 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-09T15:17:44.308 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T15:17:44.308 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T15:17:44.737 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T15:17:44.737 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:17:44.738 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:17:45.291 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-09T15:17:45.448 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-09T15:17:45.448 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-09T15:17:45.448 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-09T15:17:45.448 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-09T15:17:45.448 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-09T15:17:45.448 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-09T15:17:45.449 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-09T15:17:45.449 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-09T15:17:45.569 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-09T15:17:45.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-09T15:17:45.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-09T15:17:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:17:45.605 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-09T15:17:45.606 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-09T15:17:45.606 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-427.el9.x86_64 2024-03-09T15:17:45.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-09T15:17:46.213 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-09T15:17:47.542 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-09T15:17:47.563 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-09T15:17:47.565 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-09T15:17:47.580 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.NWEYsO7UeV 2024-03-09T15:17:47.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NWEYsO7UeV 2024-03-09T15:17:47.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.NWEYsO7UeV 2024-03-09T15:17:47.790 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.NWEYsO7UeV is 6KB 2024-03-09T15:17:47.839 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.NWEYsO7UeV 2024-03-09T15:17:47.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-09T15:17:47.924 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-09T15:17:47.924 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-09T15:17:47.925 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-09T15:17:47.925 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-09T15:17:47.926 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-09T15:17:48.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-09T15:18:18.069 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-09T15:18:18.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:18:18.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:18:36.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-09T15:18:45.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:18:45.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:18:48.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-09T15:19:00.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:19:00.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:20:00.711 DEBUG:teuthology.orchestra.remote:timed out 2024-03-09T15:20:15.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:20:15.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:20:16.057 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-09T15:20:16.474 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:20:16.474 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-09T15:20:16.474 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-09T15:20:16.490 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-427.el9.x86_64 2024-03-09T15:20:16.491 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-09T15:20:16.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-09T15:20:16.491 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-09T15:20:17.492 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-09T15:20:17.492 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-09T15:20:17.579 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-09T15:20:17.618 DEBUG:teuthology.parallel:result is None 2024-03-09T15:20:17.618 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-09T15:20:17.626 INFO:teuthology.task.internal:Creating test directory... 2024-03-09T15:20:17.626 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-09T15:20:17.643 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-09T15:20:17.650 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-09T15:20:17.656 INFO:teuthology.task.internal:Creating archive directory... 2024-03-09T15:20:17.656 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-09T15:20:17.709 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-09T15:20:17.716 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-09T15:20:17.716 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-09T15:20:17.785 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:20:17.799 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:20:17.801 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-09T15:20:17.809 INFO:teuthology.task.internal:Configuring sudo... 2024-03-09T15:20:17.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-09T15:20:17.870 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-09T15:20:17.880 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-09T15:20:17.880 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-09T15:20:17.926 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T15:20:18.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T15:20:18.063 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T15:20:18.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T15:20:18.189 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:20:18.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-09T15:20:18.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-09T15:20:18.340 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T15:20:18.624 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-09T15:20:18.632 INFO:teuthology.task.internal:Starting timer... 2024-03-09T15:20:18.632 INFO:teuthology.run_tasks:Running task pcp... 2024-03-09T15:20:18.642 INFO:teuthology.run_tasks:Running task selinux... 2024-03-09T15:20:18.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-09T15:20:18.715 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-09T15:20:18.717 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-09T15:20:18.717 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-09T15:20:18.751 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-09T15:20:18.751 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-09T15:20:18.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-09T15:20:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:20:18.819 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-09T15:20:18.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-09T15:20:18.895 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-09T15:20:18.904 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-09T15:20:18.919 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-09T15:20:18.921 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-09T15:24:30.154 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-09T15:24:30.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:24:30.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T15:24:30.244 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-09T15:24:30.321 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-09T15:24:30.322 INFO:teuthology.run_tasks:Running task clock... 2024-03-09T15:24:30.332 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-09T15:24:30.332 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-09T15:24:30.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T15:24:30.392 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-09T15:24:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-09T15:24:30.438 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-09T15:24:30.451 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-09T15:24:30.468 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-09T15:24:30.484 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-09T15:24:30.534 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-09T15:24:30.538 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T15:24:30.538 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-09T15:24:30.538 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T15:24:30.539 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T15:24:30.539 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T15:24:30.539 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T15:24:30.539 INFO:teuthology.run_tasks:Running task install... 2024-03-09T15:24:30.547 DEBUG:teuthology.task.install:project ceph 2024-03-09T15:24:30.548 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a96b1c0427a378792255b04333efc5a5569fd416'}} 2024-03-09T15:24:30.548 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a96b1c0427a378792255b04333efc5a5569fd416'} 2024-03-09T15:24:30.548 INFO:teuthology.task.install:Using flavor: default 2024-03-09T15:24:30.557 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-09T15:24:30.557 INFO:teuthology.task.install:extra packages: [] 2024-03-09T15:24:30.557 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a96b1c0427a378792255b04333efc5a5569fd416', 'tag': None, 'wait_for_package': False} 2024-03-09T15:24:30.558 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a96b1c0427a378792255b04333efc5a5569fd416 2024-03-09T15:24:30.704 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-03-01-1606/a96b1c0427a378792255b04333efc5a5569fd416/centos/9/flavors/default/ 2024-03-09T15:24:30.704 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1817.ga96b1c04 2024-03-09T15:24:30.914 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-03-01-1606/a96b1c0427a378792255b04333efc5a5569fd416/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-03-01-1606/a96b1c0427a378792255b04333efc5a5569fd416/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-03-01-1606/a96b1c0427a378792255b04333efc5a5569fd416/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-09T15:24:30.914 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:24:30.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-09T15:24:30.943 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-09T15:24:30.943 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a96b1c0427a378792255b04333efc5a5569fd416/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-09T15:24:31.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-09T15:24:31.092 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-09T15:24:31.159 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-03-09T15:24:31.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-03-09T15:24:31.406 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-03-09T15:24:31.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-03-09T15:24:32.417 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 123 kB/s | 82 kB 00:00 2024-03-09T15:24:33.025 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 18 kB/s | 10 kB 00:00 2024-03-09T15:24:33.543 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-09T15:24:34.178 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-09T15:24:35.606 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-09T15:24:37.621 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-09T15:24:39.989 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-09T15:24:44.815 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-09T15:24:46.524 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-03-09T15:24:47.625 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-09T15:24:52.500 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-09T15:24:55.028 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:24:55.040 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:24:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:24:55.040 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:24:55.040 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:24:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 9.8 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 3.4 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 3.1 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 5.3 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 20 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 25 k 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 745 k 2024-03-09T15:24:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 501 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 5.1 M 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 45 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 131 k 2024-03-09T15:24:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 162 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 321 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 298 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 99 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:24:55.043 INFO:teuthology.orchestra.run.smithi146.stdout:Install 30 Packages 2024-03-09T15:24:55.044 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-03-09T15:24:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:24:55.044 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 58 M 2024-03-09T15:24:55.044 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:24:56.251 INFO:teuthology.orchestra.run.smithi146.stdout:(1/32): ceph-radosgw-19.0.0-1817.ga96b1c04.el9. 13 MB/s | 9.8 MB 00:00 2024-03-09T15:24:56.293 INFO:teuthology.orchestra.run.smithi146.stdout:(2/32): ceph-selinux-19.0.0-1817.ga96b1c04.el9. 606 kB/s | 25 kB 00:00 2024-03-09T15:24:56.360 INFO:teuthology.orchestra.run.smithi146.stdout:(3/32): libcephfs2-19.0.0-1817.ga96b1c04.el9.x8 11 MB/s | 745 kB 00:00 2024-03-09T15:24:56.410 INFO:teuthology.orchestra.run.smithi146.stdout:(4/32): libradosstriper1-19.0.0-1817.ga96b1c04. 9.8 MB/s | 501 kB 00:00 2024-03-09T15:24:56.702 INFO:teuthology.orchestra.run.smithi146.stdout:(5/32): librgw2-19.0.0-1817.ga96b1c04.el9.x86_6 18 MB/s | 5.1 MB 00:00 2024-03-09T15:24:56.736 INFO:teuthology.orchestra.run.smithi146.stdout:(6/32): python3-ceph-argparse-19.0.0-1817.ga96b 1.3 MB/s | 45 kB 00:00 2024-03-09T15:24:56.786 INFO:teuthology.orchestra.run.smithi146.stdout:(7/32): python3-ceph-common-19.0.0-1817.ga96b1c 2.6 MB/s | 131 kB 00:00 2024-03-09T15:24:56.836 INFO:teuthology.orchestra.run.smithi146.stdout:(8/32): python3-cephfs-19.0.0-1817.ga96b1c04.el 3.2 MB/s | 162 kB 00:00 2024-03-09T15:24:56.895 INFO:teuthology.orchestra.run.smithi146.stdout:(9/32): python3-rados-19.0.0-1817.ga96b1c04.el9 5.4 MB/s | 321 kB 00:00 2024-03-09T15:24:56.953 INFO:teuthology.orchestra.run.smithi146.stdout:(10/32): python3-rbd-19.0.0-1817.ga96b1c04.el9. 5.0 MB/s | 298 kB 00:00 2024-03-09T15:24:57.012 INFO:teuthology.orchestra.run.smithi146.stdout:(11/32): python3-rgw-19.0.0-1817.ga96b1c04.el9. 1.7 MB/s | 99 kB 00:00 2024-03-09T15:24:57.146 INFO:teuthology.orchestra.run.smithi146.stdout:(12/32): ceph-base-19.0.0-1817.ga96b1c04.el9.x8 3.3 MB/s | 5.3 MB 00:01 2024-03-09T15:24:57.171 INFO:teuthology.orchestra.run.smithi146.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-03-09T15:24:57.196 INFO:teuthology.orchestra.run.smithi146.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-09T15:24:57.221 INFO:teuthology.orchestra.run.smithi146.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-09T15:24:57.255 INFO:teuthology.orchestra.run.smithi146.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-09T15:24:57.288 INFO:teuthology.orchestra.run.smithi146.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-09T15:24:57.314 INFO:teuthology.orchestra.run.smithi146.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2024-03-09T15:24:57.481 INFO:teuthology.orchestra.run.smithi146.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 209 kB/s | 35 kB 00:00 2024-03-09T15:24:57.539 INFO:teuthology.orchestra.run.smithi146.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 766 kB/s | 44 kB 00:00 2024-03-09T15:24:57.590 INFO:teuthology.orchestra.run.smithi146.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 720 kB/s | 216 kB 00:00 2024-03-09T15:24:57.915 INFO:teuthology.orchestra.run.smithi146.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 823 kB/s | 309 kB 00:00 2024-03-09T15:24:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-03-09T15:24:58.074 INFO:teuthology.orchestra.run.smithi146.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-09T15:24:58.157 INFO:teuthology.orchestra.run.smithi146.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-09T15:24:58.183 INFO:teuthology.orchestra.run.smithi146.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-03-09T15:24:58.250 INFO:teuthology.orchestra.run.smithi146.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-09T15:24:58.275 INFO:teuthology.orchestra.run.smithi146.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-09T15:24:58.408 INFO:teuthology.orchestra.run.smithi146.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-09T15:24:58.934 INFO:teuthology.orchestra.run.smithi146.stdout:(30/32): ceph-common-19.0.0-1817.ga96b1c04.el9. 5.8 MB/s | 20 MB 00:03 2024-03-09T15:24:59.067 INFO:teuthology.orchestra.run.smithi146.stdout:(31/32): librbd1-19.0.0-1817.ga96b1c04.el9.x86_ 4.8 MB/s | 3.1 MB 00:00 2024-03-09T15:24:59.342 INFO:teuthology.orchestra.run.smithi146.stdout:(32/32): librados2-19.0.0-1817.ga96b1c04.el9.x8 3.2 MB/s | 3.4 MB 00:01 2024-03-09T15:24:59.347 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:24:59.347 INFO:teuthology.orchestra.run.smithi146.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-09T15:24:59.379 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:24:59.415 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:24:59.415 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:24:59.889 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:24:59.890 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:01.356 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:01.436 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-09T15:25:01.464 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-09T15:25:01.489 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-09T15:25:01.512 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-09T15:25:01.529 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-09T15:25:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-09T15:25:01.776 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-09T15:25:02.134 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-09T15:25:02.222 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1817.ga96b1c04.el9.x86_64 7/34 2024-03-09T15:25:02.316 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1817.ga96b1c04.el9.x86_64 7/34 2024-03-09T15:25:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1817.ga96b1c04.el9.x86_64 8/34 2024-03-09T15:25:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1817.ga96b1c04.el9.x86_64 9/34 2024-03-09T15:25:02.496 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1817.ga96b1c04.el9.x86_64 9/34 2024-03-09T15:25:02.524 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-09T15:25:02.568 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1817.ga96b1c04.el9. 11/34 2024-03-09T15:25:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1817.ga96b1c04.el9.x86_64 12/34 2024-03-09T15:25:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1817.ga96b1c04.el9.x86_6 13/34 2024-03-09T15:25:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1817.ga96b1c04.el9.x86_6 13/34 2024-03-09T15:25:02.734 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-09T15:25:02.842 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-09T15:25:02.877 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-09T15:25:02.917 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-09T15:25:02.950 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-09T15:25:02.978 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-09T15:25:03.011 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-09T15:25:03.332 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-09T15:25:03.366 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1817.ga96b1c04.el9.x86_64 22/34 2024-03-09T15:25:03.456 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1817.ga96b1c04.el9.x86_64 22/34 2024-03-09T15:25:03.488 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 23/34 2024-03-09T15:25:03.524 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-09T15:25:03.973 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-09T15:25:04.002 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1817.ga96b1c04.el9.x86_64 26/34 2024-03-09T15:25:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1817.ga96b1c04.el9.x86_64 26/34 2024-03-09T15:25:04.124 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 27/34 2024-03-09T15:25:04.154 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1817.ga96b1c04.el9.x8 28/34 2024-03-09T15:25:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 29/34 2024-03-09T15:25:06.400 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 29/34 2024-03-09T15:25:06.960 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 29/34 2024-03-09T15:25:06.995 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1817.ga96b1c04.el9.x86_64 30/34 2024-03-09T15:25:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1817.ga96b1c04.el9.x86_64 30/34 2024-03-09T15:25:07.087 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-09T15:25:07.087 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-09T15:25:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1817.ga96b1c04.el9.x86_64 31/34 2024-03-09T15:25:16.588 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1817.ga96b1c04.el9.x86_64 31/34 2024-03-09T15:25:16.589 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-03-09T15:25:16.589 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-03-09T15:25:16.589 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-03-09T15:25:16.589 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-03-09T15:25:16.589 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-03-09T15:25:16.590 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-03-09T15:25:16.590 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-03-09T15:25:16.590 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:17.528 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 32/34 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 32/34 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-09T15:25:17.559 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:17.561 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-09T15:25:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-09T15:25:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 2/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 3/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1817.ga96b1c04.el9.x86_64 4/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1817.ga96b1c04.el9.x86_64 5/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1817.ga96b1c04.el9.x86_6 6/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1817.ga96b1c04.el9.x86_64 7/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1817.ga96b1c04.el9. 8/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1817.ga96b1c04.el9.x8 9/34 2024-03-09T15:25:18.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1817.ga96b1c04.el9.x86_64 10/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1817.ga96b1c04.el9.x86_64 11/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 12/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 13/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-09T15:25:18.719 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1817.ga96b1c04.el9.x86_64 31/34 2024-03-09T15:25:18.720 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-09T15:25:18.721 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1817.ga96b1c04.el9.x86_64 33/34 2024-03-09T15:25:19.787 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-09T15:25:19.788 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.789 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:19.790 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:20.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-03-09T15:25:20.668 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:28 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:20.818 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 47 M 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 163 k 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:20.819 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 48 M 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 205 M 2024-03-09T15:25:20.820 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:25:21.100 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-09T15:25:21.151 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-09T15:25:21.184 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-09T15:25:21.293 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): libcephsqlite-19.0.0-1817.ga96b1c04.el9. 465 kB/s | 163 kB 00:00 2024-03-09T15:25:21.701 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-03-09T15:25:24.893 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-19.0.0-1817.ga96b1c04.el9.x86_ 12 MB/s | 47 MB 00:03 2024-03-09T15:25:24.896 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:25:24.897 INFO:teuthology.orchestra.run.smithi146.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-09T15:25:24.911 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:25:24.933 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:25:24.933 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:25:25.332 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:25:25.332 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:25.937 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:26.027 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-09T15:25:26.067 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-09T15:25:26.108 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-09T15:25:26.151 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-09T15:25:26.180 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1817.ga96b1c04.el9.x86_64 5/6 2024-03-09T15:25:30.683 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1817.ga96b1c04.el9.x86_64 5/6 2024-03-09T15:25:30.767 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1817.ga96b1c04.el9.x86_64 6/6 2024-03-09T15:25:33.099 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:19.0.0-1817.ga96b1c04.el9.x86_64 6/6 2024-03-09T15:25:33.100 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/6 2024-03-09T15:25:33.100 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1817.ga96b1c04.el9.x86_64 2/6 2024-03-09T15:25:33.100 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-09T15:25:33.100 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-09T15:25:33.100 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-09T15:25:33.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:33.485 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:33.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-03-09T15:25:34.346 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:42 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:34.518 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 6.6 k 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 2.1 M 2024-03-09T15:25:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 1.5 M 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 248 k 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 4.7 M 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 17 M 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-09T15:25:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-09T15:25:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:25:34.524 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-09T15:25:34.525 INFO:teuthology.orchestra.run.smithi146.stdout:Install 36 Packages 2024-03-09T15:25:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:34.525 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2024-03-09T15:25:34.525 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 107 M 2024-03-09T15:25:34.525 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:25:35.254 INFO:teuthology.orchestra.run.smithi146.stdout:(1/36): ceph-19.0.0-1817.ga96b1c04.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-09T15:25:36.196 INFO:teuthology.orchestra.run.smithi146.stdout:(2/36): ceph-mds-19.0.0-1817.ga96b1c04.el9.x86_ 1.7 MB/s | 2.1 MB 00:01 2024-03-09T15:25:36.371 INFO:teuthology.orchestra.run.smithi146.stdout:(3/36): ceph-mgr-19.0.0-1817.ga96b1c04.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-03-09T15:25:36.679 INFO:teuthology.orchestra.run.smithi146.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1817.ga96b 803 kB/s | 248 kB 00:00 2024-03-09T15:25:36.796 INFO:teuthology.orchestra.run.smithi146.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-09T15:25:36.863 INFO:teuthology.orchestra.run.smithi146.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-09T15:25:36.922 INFO:teuthology.orchestra.run.smithi146.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 544 kB/s | 32 kB 00:00 2024-03-09T15:25:36.955 INFO:teuthology.orchestra.run.smithi146.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-09T15:25:36.997 INFO:teuthology.orchestra.run.smithi146.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-09T15:25:37.056 INFO:teuthology.orchestra.run.smithi146.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-09T15:25:37.131 INFO:teuthology.orchestra.run.smithi146.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-09T15:25:37.165 INFO:teuthology.orchestra.run.smithi146.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-09T15:25:37.198 INFO:teuthology.orchestra.run.smithi146.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-09T15:25:37.240 INFO:teuthology.orchestra.run.smithi146.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-09T15:25:37.465 INFO:teuthology.orchestra.run.smithi146.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 555 kB/s | 125 kB 00:00 2024-03-09T15:25:37.757 INFO:teuthology.orchestra.run.smithi146.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-09T15:25:38.083 INFO:teuthology.orchestra.run.smithi146.stdout:(17/36): ceph-osd-19.0.0-1817.ga96b1c04.el9.x86 8.9 MB/s | 17 MB 00:01 2024-03-09T15:25:38.208 INFO:teuthology.orchestra.run.smithi146.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 336 kB/s | 151 kB 00:00 2024-03-09T15:25:38.317 INFO:teuthology.orchestra.run.smithi146.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-03-09T15:25:38.342 INFO:teuthology.orchestra.run.smithi146.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 168 kB/s | 43 kB 00:00 2024-03-09T15:25:38.401 INFO:teuthology.orchestra.run.smithi146.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-09T15:25:38.425 INFO:teuthology.orchestra.run.smithi146.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-09T15:25:38.459 INFO:teuthology.orchestra.run.smithi146.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-09T15:25:38.484 INFO:teuthology.orchestra.run.smithi146.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-09T15:25:38.520 INFO:teuthology.orchestra.run.smithi146.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 320 kB/s | 19 kB 00:00 2024-03-09T15:25:38.545 INFO:teuthology.orchestra.run.smithi146.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 323 kB/s | 20 kB 00:00 2024-03-09T15:25:38.579 INFO:teuthology.orchestra.run.smithi146.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 791 kB/s | 46 kB 00:00 2024-03-09T15:25:38.629 INFO:teuthology.orchestra.run.smithi146.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-03-09T15:25:38.688 INFO:teuthology.orchestra.run.smithi146.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-09T15:25:38.763 INFO:teuthology.orchestra.run.smithi146.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-09T15:25:38.805 INFO:teuthology.orchestra.run.smithi146.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 771 kB/s | 90 kB 00:00 2024-03-09T15:25:38.872 INFO:teuthology.orchestra.run.smithi146.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 538 kB/s | 36 kB 00:00 2024-03-09T15:25:38.931 INFO:teuthology.orchestra.run.smithi146.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-09T15:25:38.956 INFO:teuthology.orchestra.run.smithi146.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-09T15:25:38.989 INFO:teuthology.orchestra.run.smithi146.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-09T15:25:39.131 INFO:teuthology.orchestra.run.smithi146.stdout:(36/36): ceph-mon-19.0.0-1817.ga96b1c04.el9.x86 1.2 MB/s | 4.7 MB 00:03 2024-03-09T15:25:39.137 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:25:39.137 INFO:teuthology.orchestra.run.smithi146.stdout:Total 6.4 MB/s | 30 MB 00:04 2024-03-09T15:25:39.161 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:25:39.204 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:25:39.204 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:25:39.600 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:25:39.601 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-09T15:25:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-09T15:25:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-09T15:25:41.126 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-09T15:25:41.168 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-09T15:25:41.201 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-09T15:25:41.293 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-09T15:25:41.382 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-09T15:25:41.420 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-09T15:25:41.493 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-09T15:25:41.546 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-09T15:25:41.584 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-09T15:25:41.633 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-09T15:25:41.688 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-09T15:25:41.880 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-09T15:25:41.938 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-09T15:25:41.976 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-09T15:25:42.011 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-09T15:25:42.041 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-09T15:25:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-09T15:25:42.151 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-09T15:25:42.184 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-09T15:25:42.225 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-09T15:25:42.260 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-09T15:25:42.294 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-09T15:25:42.385 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-09T15:25:42.425 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-09T15:25:42.499 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-09T15:25:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-09T15:25:42.760 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-09T15:25:42.926 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1817.ga96b1c04.el9. 31/36 2024-03-09T15:25:42.955 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1817.ga96b1c04.el9.x86_64 32/36 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1817.ga96b1c04.el9.x86_64 32/36 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-09T15:25:42.981 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:44.423 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1817.ga96b1c04.el9.x86_64 33/36 2024-03-09T15:25:44.448 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1817.ga96b1c04.el9.x86_64 33/36 2024-03-09T15:25:44.448 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:44.448 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-09T15:25:44.449 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-09T15:25:44.449 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-09T15:25:44.449 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:44.898 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1817.ga96b1c04.el9.x86_64 34/36 2024-03-09T15:25:44.921 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1817.ga96b1c04.el9.x86_64 34/36 2024-03-09T15:25:44.921 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:44.921 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-09T15:25:44.921 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-09T15:25:44.922 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-09T15:25:44.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:45.124 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1817.ga96b1c04.el9.x86_64 35/36 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1817.ga96b1c04.el9.x86_64 35/36 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-09T15:25:45.147 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1817.ga96b1c04.el9.x86_64 36/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:19.0.0-1817.ga96b1c04.el9.x86_64 36/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1817.ga96b1c04.el9.x86_64 2/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1817.ga96b1c04.el9.x86_64 3/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1817.ga96b1c04.el9.x86_64 4/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1817.ga96b1c04.el9.x86_64 5/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1817.ga96b1c04.el9. 6/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-09T15:25:47.061 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-09T15:25:47.062 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-09T15:25:47.063 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-09T15:25:47.064 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-09T15:25:48.236 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-09T15:25:48.236 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:48.236 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:25:48.236 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-09T15:25:48.237 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-09T15:25:48.238 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:48.239 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:48.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-03-09T15:25:49.037 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:57 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:49.134 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:25:49.186 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:49.187 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:25:49.187 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:49.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-03-09T15:25:49.808 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:57 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:49.957 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 748 k 2024-03-09T15:25:49.958 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 748 k 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 755 k 2024-03-09T15:25:49.959 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:25:50.446 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-19.0.0-1817.ga96b1c04.el9.noarch.rpm 1.5 MB/s | 748 kB 00:00 2024-03-09T15:25:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:25:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.5 MB/s | 748 kB 00:00 2024-03-09T15:25:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:25:50.452 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:25:50.453 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:25:50.458 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:25:50.459 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:50.533 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:50.990 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 1/1 2024-03-09T15:25:51.076 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 1/1 2024-03-09T15:25:51.477 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 1/1 2024-03-09T15:25:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 1/1 2024-03-09T15:25:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:51.797 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:25:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:25:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:51.797 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:51.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-03-09T15:25:52.550 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:00 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:52.704 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:52.705 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:52.705 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-09T15:25:52.705 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:52.705 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:25:52.705 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 145 k 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 145 k 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 436 k 2024-03-09T15:25:52.706 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:25:53.110 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-19.0.0-1817.ga96b1c 359 kB/s | 145 kB 00:00 2024-03-09T15:25:53.111 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:25:53.111 INFO:teuthology.orchestra.run.smithi146.stdout:Total 358 kB/s | 145 kB 00:00 2024-03-09T15:25:53.111 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:25:53.119 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:25:53.119 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:25:53.170 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:25:53.171 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1817.ga96b1c04. 1/1 2024-03-09T15:25:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1817.ga96b1c04. 1/1 2024-03-09T15:25:53.403 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:25:53.403 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-09T15:25:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1817.ga96b1c04. 1/1 2024-03-09T15:25:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:54.417 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:25:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:25:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:54.417 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:54.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-03-09T15:25:55.212 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:03 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:55.312 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:25:55.363 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:55.364 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:25:55.364 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:25:55.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-03-09T15:25:55.989 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:03 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:25:56.139 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:25:56.140 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 1.8 M 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 31 k 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 16 k 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-09T15:25:56.141 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-09T15:25:56.142 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-09T15:25:56.143 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:25:56.143 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.9 M 2024-03-09T15:25:56.143 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-09T15:25:56.144 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:25:56.713 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1817.ga9 73 kB/s | 31 kB 00:00 2024-03-09T15:25:56.739 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1817.ga96 36 kB/s | 16 kB 00:00 2024-03-09T15:25:57.047 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 835 kB/s | 238 kB 00:00 2024-03-09T15:25:57.148 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-03-09T15:25:57.198 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-03-09T15:25:57.340 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-09T15:25:57.540 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-09T15:25:57.665 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-09T15:25:57.757 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-09T15:25:57.799 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): ceph-mgr-dashboard-19.0.0-1817.ga96b1c 1.2 MB/s | 1.8 MB 00:01 2024-03-09T15:25:57.841 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.1 MB/s | 2.0 MB 00:00 2024-03-09T15:25:57.908 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-09T15:25:57.911 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:25:57.912 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-09T15:25:57.929 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:25:57.947 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:25:57.948 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:25:58.119 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:25:58.120 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:25:58.630 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:25:58.716 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-09T15:25:58.752 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-09T15:25:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-09T15:25:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-09T15:25:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-09T15:25:59.054 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-09T15:25:59.149 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-09T15:25:59.200 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-09T15:25:59.228 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-09T15:25:59.260 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1817.ga96b1c04.el9 10/12 2024-03-09T15:25:59.509 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1817.ga96b1c04.el 11/12 2024-03-09T15:25:59.543 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1817.ga96b1c04.el9.noa 12/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1817.ga96b1c04.el9.noa 12/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1817.ga96b1c04.el 1/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1817.ga96b1c04.el9.noa 2/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1817.ga96b1c04.el9 3/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-09T15:26:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-09T15:26:00.369 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-09T15:26:00.718 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-09T15:26:00.719 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-09T15:26:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-09T15:26:00.720 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:00.720 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:00.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-09T15:26:01.528 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:09 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:01.678 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:01.679 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-09T15:26:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:26:01.679 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-09T15:26:01.679 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 7.4 M 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-09T15:26:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2024-03-09T15:26:01.681 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2024-03-09T15:26:01.682 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:02.106 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-09T15:26:02.139 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-03-09T15:26:02.181 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 838 kB/s | 34 kB 00:00 2024-03-09T15:26:02.232 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-09T15:26:02.524 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-09T15:26:02.582 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-09T15:26:02.607 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-09T15:26:02.958 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-09T15:26:03.466 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-03-09T15:26:03.502 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 902 kB/s | 803 kB 00:00 2024-03-09T15:26:03.675 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-09T15:26:03.833 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.0 MB/s | 7.4 MB 00:01 2024-03-09T15:26:03.835 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:03.836 INFO:teuthology.orchestra.run.smithi146.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-09T15:26:03.876 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:03.900 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:03.900 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:04.271 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:04.272 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:04.981 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:05.056 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-09T15:26:05.087 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-09T15:26:05.182 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-09T15:26:05.232 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-09T15:26:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-09T15:26:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-09T15:26:05.522 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-09T15:26:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-09T15:26:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-09T15:26:07.498 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-09T15:26:08.515 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-09T15:26:08.555 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1817.ga96b1 12/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1817.ga96b1 12/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1817.ga96b1 1/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-09T15:26:11.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-09T15:26:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:11.524 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:11.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-03-09T15:26:12.425 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:20 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:12.575 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:12.576 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:12.576 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:26:12.576 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 49 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-09T15:26:12.577 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:12.578 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2024-03-09T15:26:12.579 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-09T15:26:12.579 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:12.748 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-09T15:26:12.772 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-09T15:26:12.823 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-09T15:26:12.848 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-09T15:26:12.865 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-09T15:26:12.890 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-09T15:26:13.049 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): ceph-mgr-rook-19.0.0-1817.ga96b1c04.el9 116 kB/s | 49 kB 00:00 2024-03-09T15:26:13.149 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-09T15:26:13.166 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-09T15:26:13.308 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-09T15:26:13.433 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-03-09T15:26:13.450 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 507 kB/s | 203 kB 00:00 2024-03-09T15:26:13.484 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-09T15:26:13.488 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:13.488 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-09T15:26:13.488 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:13.501 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:13.501 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:13.603 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:13.604 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:14.136 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:14.308 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-09T15:26:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-09T15:26:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-09T15:26:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-09T15:26:14.502 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-09T15:26:14.597 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-09T15:26:14.706 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-09T15:26:14.762 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-09T15:26:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-09T15:26:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-09T15:26:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-09T15:26:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-09T15:26:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1817.ga96b1c04.el9.noarch 13/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1817.ga96b1c04.el9.noarch 13/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1817.ga96b1c04.el9.noarch 1/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-09T15:26:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-09T15:26:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-09T15:26:16.830 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:16.831 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:17.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-03-09T15:26:17.670 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:25 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:17.821 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 138 k 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:26:17.822 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-09T15:26:17.823 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:17.824 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2024-03-09T15:26:17.824 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2024-03-09T15:26:17.824 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:18.293 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-09T15:26:18.318 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1817.ga96b1c04.e 479 kB/s | 138 kB 00:00 2024-03-09T15:26:18.351 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 770 kB/s | 248 kB 00:00 2024-03-09T15:26:18.410 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 628 kB/s | 58 kB 00:00 2024-03-09T15:26:18.435 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 611 kB/s | 51 kB 00:00 2024-03-09T15:26:18.461 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 3.2 MB/s | 548 kB 00:00 2024-03-09T15:26:18.461 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:18.462 INFO:teuthology.orchestra.run.smithi146.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-03-09T15:26:18.474 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:18.485 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:18.485 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:18.577 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:18.579 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:18.873 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:19.033 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-09T15:26:19.088 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-09T15:26:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-09T15:26:19.690 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-09T15:26:19.753 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-09T15:26:19.783 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 6/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 6/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 1/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-09T15:26:20.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-09T15:26:20.850 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-09T15:26:20.850 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:20.850 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:20.850 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:20.851 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:21.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-03-09T15:26:21.658 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:29 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:21.811 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 887 k 2024-03-09T15:26:21.812 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 966 k 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.8 M 2024-03-09T15:26:21.813 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:21.945 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 596 kB/s | 79 kB 00:00 2024-03-09T15:26:22.979 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-19.0.0-1817.ga96b1c04.el9.x86_ 762 kB/s | 887 kB 00:01 2024-03-09T15:26:22.980 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:22.981 INFO:teuthology.orchestra.run.smithi146.stdout:Total 827 kB/s | 966 kB 00:01 2024-03-09T15:26:22.981 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:23.004 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:23.004 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:23.084 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:23.084 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:23.270 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:23.413 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-09T15:26:23.480 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 2/2 2024-03-09T15:26:24.179 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 2/2 2024-03-09T15:26:24.179 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/2 2024-03-09T15:26:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-09T15:26:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:24.514 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-09T15:26:24.514 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:24.515 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:24.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2024-03-09T15:26:25.380 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:33 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:25.535 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:25.535 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:25.535 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-09T15:26:25.535 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:25.535 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1817.ga96b1c04.el9 ceph-noarch 264 k 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-09T15:26:25.536 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:25.537 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 332 k 2024-03-09T15:26:25.537 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.5 M 2024-03-09T15:26:25.537 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:25.658 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 569 kB/s | 69 kB 00:00 2024-03-09T15:26:25.824 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-volume-19.0.0-1817.ga96b1c04.el9.no 917 kB/s | 264 kB 00:00 2024-03-09T15:26:25.825 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:25.826 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-09T15:26:25.826 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:25.834 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:25.835 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:25.878 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:25.879 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:26.030 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:26.207 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-09T15:26:26.244 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1817.ga96b1c04.el9.noarch 2/2 2024-03-09T15:26:26.259 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1817.ga96b1c04.el9.noarch 2/2 2024-03-09T15:26:26.259 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:26:26.259 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-09T15:26:26.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:27.005 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1817.ga96b1c04.el9.noarch 1/2 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1817.ga96b1c04.el9.noarch 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-09T15:26:27.364 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:27.365 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:27.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-03-09T15:26:28.329 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 127 k 2024-03-09T15:26:28.480 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 127 k 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2024-03-09T15:26:28.481 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:28.688 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-19.0.0-1817.ga96b1c04.el9.x86_64 612 kB/s | 127 kB 00:00 2024-03-09T15:26:28.689 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:28.689 INFO:teuthology.orchestra.run.smithi146.stdout:Total 608 kB/s | 127 kB 00:00 2024-03-09T15:26:28.689 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:28.695 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:28.695 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:28.751 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:28.751 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:29.004 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:29.509 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:29.797 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:26:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:29.798 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:30.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-03-09T15:26:30.612 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:38 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:30.710 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:30.766 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:30.767 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:30.767 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:30.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-03-09T15:26:31.407 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:39 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:31.570 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 31 k 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:31.571 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 140 k 2024-03-09T15:26:31.572 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:31.791 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-19.0.0-1817.ga96b1c04.el9.x86_6 142 kB/s | 31 kB 00:00 2024-03-09T15:26:31.792 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:31.792 INFO:teuthology.orchestra.run.smithi146.stdout:Total 141 kB/s | 31 kB 00:00 2024-03-09T15:26:31.792 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:31.795 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:31.795 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:31.824 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:31.824 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:31.943 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:32.031 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:32.435 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:32.724 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:32.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-03-09T15:26:33.452 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:41 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:33.552 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:33.604 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:33.605 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:33.605 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:33.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-03-09T15:26:34.211 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:34.309 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:34.370 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:34.370 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:34.370 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:34.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-03-09T15:26:34.965 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:35.061 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:35.114 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:35.114 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:35.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-03-09T15:26:35.724 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:43 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:35.821 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:35.872 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:35.873 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:35.873 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:35.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-03-09T15:26:36.486 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:44 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:36.584 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:36.636 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:36.637 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:36.637 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:36.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-03-09T15:26:37.244 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:45 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:37.342 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 is already installed. 2024-03-09T15:26:37.393 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:37.394 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-09T15:26:37.395 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:37.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-03-09T15:26:38.002 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:46 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:38.152 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:38.153 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:38.153 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-09T15:26:38.153 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 85 k 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 85 k 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 231 k 2024-03-09T15:26:38.154 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:38.444 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-19.0.0-1817.ga96b1c04.el9.x86_64.rpm 295 kB/s | 85 kB 00:00 2024-03-09T15:26:38.445 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:38.445 INFO:teuthology.orchestra.run.smithi146.stdout:Total 293 kB/s | 85 kB 00:00 2024-03-09T15:26:38.445 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:38.452 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:38.452 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:38.508 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:38.508 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:39.161 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:39.463 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:39.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-03-09T15:26:40.270 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:48 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:40.427 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:40.427 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:40.427 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 3.1 M 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.1 M 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 13 M 2024-03-09T15:26:40.428 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:41.749 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-19.0.0-1817.ga96b1c04.el9.x86_64.rpm 2.3 MB/s | 3.1 MB 00:01 2024-03-09T15:26:41.750 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:41.750 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-03-09T15:26:41.754 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:41.761 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:41.761 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:41.838 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:41.838 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:42.268 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:42.324 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-09T15:26:42.349 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:43.513 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:43.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-03-09T15:26:44.340 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:52 ago on Sat 09 Mar 2024 03:24:52 PM UTC. 2024-03-09T15:26:44.492 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-09T15:26:44.492 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:44.492 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-09T15:26:44.492 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1817.ga96b1c04.el9 ceph 170 k 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:44.493 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 170 k 2024-03-09T15:26:44.494 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 490 k 2024-03-09T15:26:44.494 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-09T15:26:44.918 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-19.0.0-1817.ga96b1c04.el9.x86_64.rpm 402 kB/s | 170 kB 00:00 2024-03-09T15:26:44.919 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-09T15:26:44.919 INFO:teuthology.orchestra.run.smithi146.stdout:Total 400 kB/s | 170 kB 00:00 2024-03-09T15:26:44.919 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-09T15:26:44.927 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-09T15:26:44.927 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-09T15:26:44.984 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-09T15:26:44.984 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-09T15:26:45.132 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-09T15:26:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:45.750 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 1/1 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1817.ga96b1c04.el9.x86_64 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-09T15:26:46.071 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-09T15:26:46.297 DEBUG:teuthology.parallel:result is None 2024-03-09T15:26:46.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a96b1c0427a378792255b04333efc5a5569fd416 2024-03-09T15:26:46.518 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-09T15:26:46.539 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1817.ga96b1c04.el9 2024-03-09T15:26:46.540 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1817.ga96b1c04.el9 2024-03-09T15:26:46.540 INFO:teuthology.task.install:The correct ceph version 19.0.0-1817.ga96b1c04 is installed. 2024-03-09T15:26:46.542 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-09T15:26:46.543 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:26:46.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-09T15:26:46.608 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-09T15:26:46.608 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:26:46.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-09T15:26:46.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-09T15:26:46.744 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-09T15:26:46.744 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:26:46.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-09T15:26:46.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-09T15:26:46.895 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-09T15:26:46.895 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:26:46.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-09T15:26:46.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-09T15:26:47.029 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T15:26:47.043 INFO:tasks.workunit:Pulling workunits from ref a96b1c0427a378792255b04333efc5a5569fd416 2024-03-09T15:26:47.044 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T15:26:47.044 INFO:tasks.workunit:timeout=3h 2024-03-09T15:26:47.044 INFO:tasks.workunit:cleanup=True 2024-03-09T15:26:47.044 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T15:26:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T15:26:47.085 INFO:teuthology.orchestra.run.smithi146.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T15:26:47.085 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T15:26:47.141 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T15:26:47.142 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T15:26:47.196 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a96b1c0427a378792255b04333efc5a5569fd416 2024-03-09T15:26:47.252 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T15:27:33.318 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to 'a96b1c0427a378792255b04333efc5a5569fd416'. 2024-03-09T15:27:33.318 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.318 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T15:27:33.318 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T15:27:33.318 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-03-09T15:27:33.319 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at a96b1c0427a Merge branch 'wip-fix-ec-inconsisten-hinfo-wrn' of https://github.com/sseshasa/ceph into wip-yuri2-testing-2024-03-01-1606 2024-03-09T15:27:33.322 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T15:27:33.379 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-09T15:27:33.379 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T15:27:33.435 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-03-09T15:27:33.435 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-03-09T15:27:33.435 DEBUG:teuthology.orchestra.run.smithi146:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-09T15:27:33.508 INFO:tasks.workunit.client.0.smithi146.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T15:27:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-03-09T15:27:33.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-03-09T15:27:33.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none ' 2024-03-09T15:27:33.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-03-09T15:27:33.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T15:27:33.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-03-09T15:27:33.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:27:33.520 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:27:33.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:27:33.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:27:33.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:27:33.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:27:33.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:27:33.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:27:33.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:27:33.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:27:33.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:27:33.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:27:33.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:27:33.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:27:33.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:27:33.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:27:33.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:27:33.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:27:33.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T15:27:33.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:27:33.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:27:33.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:27:33.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28442 2024-03-09T15:27:33.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:27:33.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:27:33.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-03-09T15:27:33.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:27:33.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:27:33.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:27:33.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28442 2024-03-09T15:27:33.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-03-09T15:27:33.531 INFO:tasks.workunit.client.0.smithi146.stdout:Dir: td/osd-mapper 2024-03-09T15:27:33.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-09T15:27:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-09T15:27:35.107 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 ec990f73-4e8e-478d-b8d5-6a61e00d77b1 2024-03-09T15:27:35.355 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:27:35.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.438+0000 7f152ba0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:35.474 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.474+0000 7f152ba0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:35.475 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.475+0000 7f152ba0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:35.476 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.476+0000 7f152ba0c640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T15:27:35.476 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.476+0000 7f152ba0c640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T15:27:35.476 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.476+0000 7f152ba0c640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T15:27:35.532 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:35.532+0000 7f152ba0c640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-03-09T15:27:41.053 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:27:41.393 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:27:41.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:41.421+0000 7f6c00fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:41.423 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:41.423+0000 7f6c00fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:41.424 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:41.424+0000 7f6c00fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:27:41.711 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:27:41.939 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:41.939+0000 7f6c00fbd640 -1 Falling back to public interface 2024-03-09T15:27:42.986 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:27:44.150 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:27:44.150+0000 7f6c00fbd640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:27:44.269 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:27:45.605 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:27:46.882 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:27:48.159 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:27:49.443 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:27:50.726 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:27:52.085 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:27:53.364 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:27:54.694 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:27:56.199 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:27:57.701 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:27:59.325 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:28:00.872 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:28:02.515 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:28:05.041 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:28:07.045 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:28:08.345 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:28:08.622 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1285034875,v1:127.0.0.1:6803/1285034875] [v2:127.0.0.1:6804/1285034875,v1:127.0.0.1:6805/1285034875] exists,up ec990f73-4e8e-478d-b8d5-6a61e00d77b1 2024-03-09T15:28:08.625 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 4c99ae61-9a84-42d0-8d8f-5138063a1025 2024-03-09T15:28:08.986 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:28:09.028 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.028+0000 7f5966373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:09.064 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.064+0000 7f5966373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:09.064 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.065+0000 7f5966373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:09.065 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.065+0000 7f5966373640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T15:28:09.065 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.065+0000 7f5966373640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T15:28:09.065 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.065+0000 7f5966373640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T15:28:09.137 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:09.137+0000 7f5966373640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-03-09T15:28:14.827 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:28:15.192 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:28:15.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:15.220+0000 7fb8f260c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:15.242 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:15.242+0000 7fb8f260c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:15.243 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:15.243+0000 7fb8f260c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:15.462 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:28:15.765 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:15.764+0000 7fb8f260c640 -1 Falling back to public interface 2024-03-09T15:28:16.741 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:28:18.021 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:28:18.108 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:18.107+0000 7fb8f260c640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:28:19.317 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:28:20.707 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:28:22.075 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:28:23.352 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:28:24.637 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:28:25.919 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:28:27.196 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:28:28.481 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:28:29.761 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:28:31.161 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:28:32.511 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:28:33.843 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:28:35.545 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:28:37.281 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:28:38.818 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:28:40.267 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:28:41.656 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:28:41.928 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2701734687,v1:127.0.0.1:6811/2701734687] [v2:127.0.0.1:6812/2701734687,v1:127.0.0.1:6813/2701734687] exists,up 4c99ae61-9a84-42d0-8d8f-5138063a1025 2024-03-09T15:28:41.932 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 14001c63-1fe1-4ae0-a4ce-6d8b246a05ec 2024-03-09T15:28:42.312 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:28:42.356 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.356+0000 7ff909d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:42.397 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.397+0000 7ff909d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:42.398 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.398+0000 7ff909d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:42.398 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.399+0000 7ff909d38640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T15:28:42.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.399+0000 7ff909d38640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T15:28:42.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.399+0000 7ff909d38640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T15:28:42.464 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:42.464+0000 7ff909d38640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-03-09T15:28:48.228 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:28:48.581 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:28:48.610 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:48.610+0000 7f488d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:48.618+0000 7f488d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:48.619+0000 7f488d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:28:48.879 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:28:49.151 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:49.151+0000 7f488d019640 -1 Falling back to public interface 2024-03-09T15:28:50.147 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:28:51.427 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:28:51.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:28:51.586+0000 7f488d019640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:28:52.770 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:28:54.049 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:28:55.347 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:28:56.703 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:28:57.985 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:28:59.365 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:29:00.670 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:29:01.955 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:29:03.236 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:29:04.743 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:29:06.403 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:29:08.063 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:29:09.741 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:29:11.478 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:29:12.967 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:29:14.781 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:29:15.062 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1397030575,v1:127.0.0.1:6819/1397030575] [v2:127.0.0.1:6820/1397030575,v1:127.0.0.1:6821/1397030575] exists,up 14001c63-1fe1-4ae0-a4ce-6d8b246a05ec 2024-03-09T15:29:15.526 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'test' created 2024-03-09T15:29:17.399 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836497 2024-03-09T15:29:19.057 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672970 2024-03-09T15:29:19.357 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509443 2024-03-09T15:29:21.842 INFO:tasks.workunit.client.0.smithi146.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-03-09T15:29:21.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-03-09T15:29:21.972 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: { 2024-03-09T15:29:21.972 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:21.972 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:21.986 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: { 2024-03-09T15:29:21.986 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:21.986 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:22.002 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2: { 2024-03-09T15:29:22.002 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:22.002 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:22.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-03-09T15:29:22.114 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: { 2024-03-09T15:29:22.114 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:22.114 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:22.128 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: { 2024-03-09T15:29:22.128 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:22.128 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:22.142 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2: { 2024-03-09T15:29:22.142 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T15:29:22.142 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T15:29:22.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T15:29:22.522 INFO:tasks.workunit.client.0.smithi146.stdout:created pool test snap snap01 2024-03-09T15:29:22.774 INFO:tasks.workunit.client.0.smithi146.stdout:created pool test snap snap02 2024-03-09T15:29:22.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-03-09T15:29:22.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T15:29:22.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T15:29:23.327 INFO:tasks.workunit.client.0.smithi146.stdout:created pool test snap snap13 2024-03-09T15:29:23.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-03-09T15:29:23.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T15:29:23.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T15:29:23.771 INFO:tasks.workunit.client.0.smithi146.stdout:created pool test snap snap24 2024-03-09T15:29:23.987 INFO:tasks.workunit.client.0.smithi146.stdout:created pool test snap snap25 2024-03-09T15:29:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 5258 2024-03-09T15:29:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-03-09T15:29:24.309 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T15:29:24.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-03-09T15:29:24.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-03-09T15:29:24.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T15:29:24.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-09T15:29:24.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-03-09T15:29:24.895 INFO:tasks.workunit.client.0.smithi146.stdout:pgid is 1.3 (primary: osd.1) 2024-03-09T15:29:24.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-03-09T15:29:24.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-03-09T15:29:24.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-03-09T15:29:24.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-03-09T15:29:24.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-03-09T15:29:25.149 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs_brief 2024-03-09T15:29:25.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-03-09T15:29:25.160 INFO:tasks.workunit.client.0.smithi146.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28442/ceph-osd.1.asok 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-03-09T15:29:25.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28442/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-03-09T15:29:25.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:25.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout:{"success":true}{ 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "objxxx", 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "clones": [ 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 2, 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 1, 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap01" 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.279 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 2, 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap02" 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 3, 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 3, 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap13" 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:29:25.280 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 5, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 4, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap24" 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 5, 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap25" 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:29:25.281 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: "id": "head", 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [], 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 5 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout: ] 2024-03-09T15:29:25.282 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:25.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T15:29:25.374 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:29:25.374 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:29:25.374 INFO:tasks.workunit.client.0.smithi146.stdout: "must": true, 2024-03-09T15:29:25.374 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "0.000000" 2024-03-09T15:29:25.374 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:29:25.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-03-09T15:29:28.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T15:29:26.083884+0000 28'7 28:46 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-09T15:29:26.083712+0000 28'7 2024-03-09T15:29:26.083712+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:29:26.083712+0000 4 0 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:17.341867+0000 0'0 28:34 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:16.239826+0000 0'0 28:31 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:16.968419+0000 0'0 28:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:29:28.658 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:29:28.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-03-09T15:29:28.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T15:29:26.083884+0000 28'7 28:46 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-09T15:29:26.083712+0000 28'7 2024-03-09T15:29:26.083712+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:29:26.083712+0000 4 0 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:17.341867+0000 0'0 28:34 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:16.239826+0000 0'0 28:31 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:29:16.968419+0000 0'0 28:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:29:28.921 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:28.922 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:29:28.922 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:29:28.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-03-09T15:29:29.277 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T15:29:29.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-03-09T15:29:29.653 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T15:29:29.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-03-09T15:29:34.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-03-09T15:29:34.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-09T15:29:34.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:29:34.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:29:34.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:29:34.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:29:34.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:29:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:29:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T15:29:35.001 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bdev(0x5634b8a6c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:35.001 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 0 bdev(0x5634b8a6c000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bdev(0x5634b8a6c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 0 bdev(0x5634b8a6c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T15:29:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.000+0000 7f9df1448980 1 bluefs mount 2024-03-09T15:29:35.011 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.010+0000 7f9df1448980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:35.036 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.035+0000 7f9df1448980 1 bluefs mount shared_bdev_used = 0 2024-03-09T15:29:35.036 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.035+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:35.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.037+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:35.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.037+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:35.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:35.040 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.038+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:35.073 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:35.073 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-09T15:29:35.073 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 freelist init 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 freelist _read_cfg 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.072+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T15:29:35.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.073+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.39236e-07 2024-03-09T15:29:35.075 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.073+0000 7f9df1448980 1 bluefs umount 2024-03-09T15:29:35.075 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.073+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) close 2024-03-09T15:29:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 0 bdev(0x5634b8a6c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T15:29:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 1 bluefs mount 2024-03-09T15:29:35.278 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.277+0000 7f9df1448980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:35.283 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.282+0000 7f9df1448980 1 bluefs mount shared_bdev_used = 27459584 2024-03-09T15:29:35.283 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.282+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:35.286 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.284+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:35.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:35.285+0000 7f9df1448980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:36.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.036+0000 7f9df1448980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:36.038 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T15:29:36.038 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 0b 05 58 02 00 00 fa 69 db 20 4d 7d 49 65 82 fc |..X....i. M}Ie..| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 b5 07 71 0c 38 77 00 00 00 00 1e 00 00 00 00 00 |..q.8w..........| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:* 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T15:29:36.039 INFO:tasks.workunit.client.0.smithi146.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T15:29:36.040 INFO:tasks.workunit.client.0.smithi146.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T15:29:36.041 INFO:tasks.workunit.client.0.smithi146.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 ec 99 0f 73 4e 8e |re...........sN.| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000230 47 8d b8 d5 6a 61 e0 0d 77 b1 00 00 00 00 00 00 |G...ja..w.......| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000240 00 00 1e 00 00 00 06 80 ec 65 61 62 97 36 01 00 |.........eab.6..| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:0000025e 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T15:29:36.042 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:0000003f 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:36.044 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.045 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:36.046 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.037+0000 7f9df1448980 1 bluefs umount 2024-03-09T15:29:36.047 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.037+0000 7f9df1448980 1 bdev(0x5634b8a6c700 td/osd-mapper/0/block) close 2024-03-09T15:29:36.274 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.274+0000 7f9df1448980 1 freelist shutdown 2024-03-09T15:29:36.274 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.274+0000 7f9df1448980 1 bdev(0x5634b8a6c000 td/osd-mapper/0/block) close 2024-03-09T15:29:36.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:36.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T15:29:36.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-03-09T15:29:36.543 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bdev(0x564769e34000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:36.543 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 0 bdev(0x564769e34000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bdev(0x564769e34000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 0 bdev(0x564769e34700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T15:29:36.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.543+0000 7f7c3af5b980 1 bluefs mount 2024-03-09T15:29:36.551 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.550+0000 7f7c3af5b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:36.559 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.558+0000 7f7c3af5b980 1 bluefs mount shared_bdev_used = 0 2024-03-09T15:29:36.559 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.558+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.560+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:36.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.561+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:36.594 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:36.594 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2024-03-09T15:29:36.594 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 freelist init 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 freelist _read_cfg 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.593+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.594+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 4.95912e-07 2024-03-09T15:29:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.594+0000 7f7c3af5b980 1 bluefs umount 2024-03-09T15:29:36.596 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.594+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) close 2024-03-09T15:29:36.809 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:36.809 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 0 bdev(0x564769e34700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:36.809 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:36.810 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T15:29:36.810 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 1 bluefs mount 2024-03-09T15:29:36.810 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.809+0000 7f7c3af5b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:36.815 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.814+0000 7f7c3af5b980 1 bluefs mount shared_bdev_used = 27459584 2024-03-09T15:29:36.815 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.814+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.816+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.816+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.816+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:36.817+0000 7f7c3af5b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:37.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:37.698+0000 7f7c3af5b980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:37.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:37.700+0000 7f7c3af5b980 1 bluefs umount 2024-03-09T15:29:37.700 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:37.700+0000 7f7c3af5b980 1 bdev(0x564769e34700 td/osd-mapper/2/block) close 2024-03-09T15:29:37.936 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:37.936+0000 7f7c3af5b980 1 freelist shutdown 2024-03-09T15:29:37.936 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:37.936+0000 7f7c3af5b980 1 bdev(0x564769e34000 td/osd-mapper/2/block) close 2024-03-09T15:29:38.059 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:38.059 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:38.060 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:38.060 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:38.060 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:38.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:38.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-03-09T15:29:38.061 INFO:tasks.workunit.client.0.smithi146.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-03-09T15:29:38.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:38.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T15:29:38.701 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T15:29:38.701 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 0b 05 58 02 00 00 fa 69 db 20 4d 7d 49 65 82 fc |..X....i. M}Ie..| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 b5 07 71 0c 38 77 02 00 00 00 1e 00 00 00 00 00 |..q.8w..........| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:* 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T15:29:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T15:29:38.703 INFO:tasks.workunit.client.0.smithi146.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 14 00 1c 63 1f e1 |re...........c..| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000230 4a e0 a4 ce 6d 8b 24 6a 05 ec 00 00 00 00 00 00 |J...m.$j........| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000240 00 00 1e 00 00 00 49 80 ec 65 3e a7 d6 34 01 00 |......I..e>..4..| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:0000025e 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:38.704 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:0000003f 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T15:29:38.705 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:38.706 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:38.707 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T15:29:38.708 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:39.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T15:29:39.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-09T15:29:39.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:29:39.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-03-09T15:29:40.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-03-09T15:29:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T15:29:40.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-03-09T15:29:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:29:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-03-09T15:29:40.396 INFO:tasks.workunit.client.0.smithi146.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-03-09T15:29:40.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-03-09T15:29:40.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T15:29:40.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T15:29:40.421 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bdev(0x560f862e6000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 0 bdev(0x560f862e6000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bdev(0x560f862e6000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 0 bdev(0x560f862e6700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T15:29:40.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.421+0000 7fc91ad6d980 1 bluefs mount 2024-03-09T15:29:40.426 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.427+0000 7fc91ad6d980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:40.435 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.435+0000 7fc91ad6d980 1 bluefs mount shared_bdev_used = 0 2024-03-09T15:29:40.435 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.436+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:40.439 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.438+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:40.443 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:40.443 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-09T15:29:40.443 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 freelist init 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 freelist _read_cfg 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.443+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.444+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.39236e-07 2024-03-09T15:29:40.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.444+0000 7fc91ad6d980 1 bluefs umount 2024-03-09T15:29:40.445 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.444+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) close 2024-03-09T15:29:40.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.687+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T15:29:40.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.687+0000 7fc91ad6d980 0 bdev(0x560f862e6700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T15:29:40.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.687+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:40.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.687+0000 7fc91ad6d980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T15:29:40.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.687+0000 7fc91ad6d980 1 bluefs mount 2024-03-09T15:29:40.693 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.693+0000 7fc91ad6d980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:40.701 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.701+0000 7fc91ad6d980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T15:29:40.701 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.701+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:40.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:40.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.704+0000 7fc91ad6d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:40.947 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.947+0000 7fc91ad6d980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:40.948 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T15:29:40.948 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 0b 05 58 02 00 00 fa 69 db 20 4d 7d 49 65 82 fc |..X....i. M}Ie..| 2024-03-09T15:29:40.948 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 b5 07 71 0c 38 77 00 00 00 00 1e 00 00 00 00 00 |..q.8w..........| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:* 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T15:29:40.949 INFO:tasks.workunit.client.0.smithi146.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T15:29:40.950 INFO:tasks.workunit.client.0.smithi146.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 ec 99 0f 73 4e 8e |re...........sN.| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000230 47 8d b8 d5 6a 61 e0 0d 77 b1 00 00 00 00 00 00 |G...ja..w.......| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000240 00 00 1e 00 00 00 06 80 ec 65 61 62 97 36 01 00 |.........eab.6..| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:0000025e 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T15:29:40.951 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout:0000003f 2024-03-09T15:29:40.952 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:40.953 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:40.954 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.955 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T15:29:40.956 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.956 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T15:29:40.956 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:40.956 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.948+0000 7fc91ad6d980 1 bluefs umount 2024-03-09T15:29:40.956 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:40.948+0000 7fc91ad6d980 1 bdev(0x560f862e6700 td/osd-mapper/0/block) close 2024-03-09T15:29:41.192 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:41.192+0000 7fc91ad6d980 1 freelist shutdown 2024-03-09T15:29:41.192 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:41.192+0000 7fc91ad6d980 1 bdev(0x560f862e6000 td/osd-mapper/0/block) close 2024-03-09T15:29:41.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T15:29:41.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T15:29:41.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:29:42.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:42.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:42.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T15:29:42.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T15:29:42.234 INFO:tasks.workunit.client.0.smithi146.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:42.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T15:29:42.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.CTH74XdpnZ_the_val 2024-03-09T15:29:42.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:42.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.CTH74XdpnZ_the_val' 2024-03-09T15:29:42.237 INFO:tasks.workunit.client.0.smithi146.stdout:Value dumped in: /tmp/tmp.CTH74XdpnZ_the_val 2024-03-09T15:29:42.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.CTH74XdpnZ_the_val 2024-03-09T15:29:42.749 INFO:tasks.workunit.client.0.smithi146.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T15:29:43.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:43.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.CTH74XdpnZ_the_val 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T15:29:43.046 INFO:tasks.workunit.client.0.smithi146.stdout:0000073 2024-03-09T15:29:43.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T15:29:43.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:44.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.CTH74XdpnZ_the_val 2024-03-09T15:29:45.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:45.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:29:45.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-09T15:29:45.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-03-09T15:29:45.138 INFO:tasks.workunit.client.0.smithi146.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-03-09T15:29:45.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T15:29:45.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T15:29:45.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bdev(0x564cf9962000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T15:29:45.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 0 bdev(0x564cf9962000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T15:29:45.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bdev(0x564cf9962000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:45.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T15:29:45.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T15:29:45.162 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 0 bdev(0x564cf9962700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T15:29:45.162 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:45.162 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-09T15:29:45.162 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.161+0000 7fb85902a980 1 bluefs mount 2024-03-09T15:29:45.171 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.171+0000 7fb85902a980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:45.211 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.211+0000 7fb85902a980 1 bluefs mount shared_bdev_used = 0 2024-03-09T15:29:45.212 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.211+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.213+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:45.215 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:45.216 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:45.216 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:45.216 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:45.216 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.214+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:45.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:45.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-03-09T15:29:45.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-03-09T15:29:45.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T15:29:45.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-03-09T15:29:45.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 freelist init 2024-03-09T15:29:45.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 freelist _read_cfg 2024-03-09T15:29:45.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.219+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T15:29:45.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.220+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.72206e-07 2024-03-09T15:29:45.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.220+0000 7fb85902a980 1 bluefs umount 2024-03-09T15:29:45.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.220+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) close 2024-03-09T15:29:45.423 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.423+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T15:29:45.423 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.423+0000 7fb85902a980 0 bdev(0x564cf9962700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T15:29:45.423 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.423+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:45.424 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.423+0000 7fb85902a980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-09T15:29:45.424 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.423+0000 7fb85902a980 1 bluefs mount 2024-03-09T15:29:45.429 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.429+0000 7fb85902a980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:45.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.437+0000 7fb85902a980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T15:29:45.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.437+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.440+0000 7fb85902a980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:45.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.671+0000 7fb85902a980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:45.673 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T15:29:45.673 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 0b 05 58 02 00 00 fa 69 db 20 4d 7d 49 65 82 fc |..X....i. M}Ie..| 2024-03-09T15:29:45.673 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 b5 07 71 0c 38 77 01 00 00 00 1e 00 00 00 00 00 |..q.8w..........| 2024-03-09T15:29:45.673 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:* 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T15:29:45.674 INFO:tasks.workunit.client.0.smithi146.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T15:29:45.675 INFO:tasks.workunit.client.0.smithi146.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 4c 99 ae 61 9a 84 |re........L..a..| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000230 42 d0 8d 8f 51 38 06 3a 10 25 00 00 00 00 00 00 |B...Q8.:.%......| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000240 00 00 1e 00 00 00 28 80 ec 65 09 40 d6 1c 01 00 |......(..e.@....| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:0000025e 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:45.676 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:0000003f 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:45.677 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:45.678 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:45.679 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:45.680 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:45.681 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.672+0000 7fb85902a980 1 bluefs umount 2024-03-09T15:29:45.682 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.672+0000 7fb85902a980 1 bdev(0x564cf9962700 td/osd-mapper/1/block) close 2024-03-09T15:29:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.911+0000 7fb85902a980 1 freelist shutdown 2024-03-09T15:29:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:45.911+0000 7fb85902a980 1 bdev(0x564cf9962000 td/osd-mapper/1/block) close 2024-03-09T15:29:45.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T15:29:45.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T15:29:45.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:29:46.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:46.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:46.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T15:29:46.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T15:29:46.722 INFO:tasks.workunit.client.0.smithi146.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:46.722 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T15:29:46.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.uJNLP43IIa_the_val 2024-03-09T15:29:46.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:46.723 INFO:tasks.workunit.client.0.smithi146.stdout:Value dumped in: /tmp/tmp.uJNLP43IIa_the_val 2024-03-09T15:29:46.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.uJNLP43IIa_the_val' 2024-03-09T15:29:46.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.uJNLP43IIa_the_val 2024-03-09T15:29:47.290 INFO:tasks.workunit.client.0.smithi146.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T15:29:47.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:47.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.uJNLP43IIa_the_val 2024-03-09T15:29:47.766 INFO:tasks.workunit.client.0.smithi146.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T15:29:47.766 INFO:tasks.workunit.client.0.smithi146.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T15:29:47.766 INFO:tasks.workunit.client.0.smithi146.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stdout:0000073 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T15:29:47.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:48.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.uJNLP43IIa_the_val 2024-03-09T15:29:49.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:49.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:29:49.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-03-09T15:29:49.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-03-09T15:29:49.637 INFO:tasks.workunit.client.0.smithi146.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-03-09T15:29:49.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T15:29:49.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T15:29:49.638 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bdev(0x55a59b572000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 0 bdev(0x55a59b572000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bdev(0x55a59b572000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 0 bdev(0x55a59b572700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T15:29:49.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.627+0000 7fb172210980 1 bluefs mount 2024-03-09T15:29:49.641 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.641+0000 7fb172210980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:49.650 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.650+0000 7fb172210980 1 bluefs mount shared_bdev_used = 0 2024-03-09T15:29:49.650 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.650+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:49.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:49.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:49.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:49.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:49.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.652+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.653+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.653+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.653+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.653+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.653+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.657+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.657+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.657+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.657+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.657+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-09T15:29:49.658 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 freelist init 2024-03-09T15:29:49.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 freelist _read_cfg 2024-03-09T15:29:49.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T15:29:49.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 4.95912e-07 2024-03-09T15:29:49.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 bluefs umount 2024-03-09T15:29:49.660 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.658+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) close 2024-03-09T15:29:49.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.890+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T15:29:49.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.890+0000 7fb172210980 0 bdev(0x55a59b572700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T15:29:49.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.890+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T15:29:49.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.890+0000 7fb172210980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T15:29:49.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.890+0000 7fb172210980 1 bluefs mount 2024-03-09T15:29:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.899+0000 7fb172210980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T15:29:49.907 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.907+0000 7fb172210980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T15:29:49.908 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.907+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.909+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T15:29:49.912 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T15:29:49.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:49.910+0000 7fb172210980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T15:29:50.128 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:50.127+0000 7fb172210980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T15:29:50.131 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 0b 05 58 02 00 00 fa 69 db 20 4d 7d 49 65 82 fc |..X....i. M}Ie..| 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 b5 07 71 0c 38 77 02 00 00 00 1e 00 00 00 00 00 |..q.8w..........| 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:* 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T15:29:50.132 INFO:tasks.workunit.client.0.smithi146.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T15:29:50.133 INFO:tasks.workunit.client.0.smithi146.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T15:29:50.133 INFO:tasks.workunit.client.0.smithi146.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T15:29:50.133 INFO:tasks.workunit.client.0.smithi146.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T15:29:50.133 INFO:tasks.workunit.client.0.smithi146.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T15:29:50.134 INFO:tasks.workunit.client.0.smithi146.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T15:29:50.135 INFO:tasks.workunit.client.0.smithi146.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T15:29:50.135 INFO:tasks.workunit.client.0.smithi146.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T15:29:50.135 INFO:tasks.workunit.client.0.smithi146.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T15:29:50.135 INFO:tasks.workunit.client.0.smithi146.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T15:29:50.135 INFO:tasks.workunit.client.0.smithi146.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T15:29:50.136 INFO:tasks.workunit.client.0.smithi146.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T15:29:50.137 INFO:tasks.workunit.client.0.smithi146.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 14 00 1c 63 1f e1 |re...........c..| 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:00000230 4a e0 a4 ce 6d 8b 24 6a 05 ec 00 00 00 00 00 00 |J...m.$j........| 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:00000240 00 00 1e 00 00 00 49 80 ec 65 3e a7 d6 34 01 00 |......I..e>..4..| 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:0000025e 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.138 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:50.139 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout:0000003f 2024-03-09T15:29:50.140 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout:00000047 2024-03-09T15:29:50.141 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:50.142 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.143 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:0000003b 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T15:29:50.144 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.145 INFO:tasks.workunit.client.0.smithi146.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T15:29:50.145 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:29:50.145 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:50.128+0000 7fb172210980 1 bluefs umount 2024-03-09T15:29:50.145 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:50.128+0000 7fb172210980 1 bdev(0x55a59b572700 td/osd-mapper/2/block) close 2024-03-09T15:29:50.356 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:50.356+0000 7fb172210980 1 freelist shutdown 2024-03-09T15:29:50.356 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:50.356+0000 7fb172210980 1 bdev(0x55a59b572000 td/osd-mapper/2/block) close 2024-03-09T15:29:50.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T15:29:50.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T15:29:50.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:29:51.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:51.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:51.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T15:29:51.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T15:29:51.184 INFO:tasks.workunit.client.0.smithi146.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:51.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T15:29:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.dpg8S8AmsU_the_val 2024-03-09T15:29:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:51.186 INFO:tasks.workunit.client.0.smithi146.stdout:Value dumped in: /tmp/tmp.dpg8S8AmsU_the_val 2024-03-09T15:29:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.dpg8S8AmsU_the_val' 2024-03-09T15:29:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.dpg8S8AmsU_the_val 2024-03-09T15:29:51.671 INFO:tasks.workunit.client.0.smithi146.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T15:29:51.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:51.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.dpg8S8AmsU_the_val 2024-03-09T15:29:51.980 INFO:tasks.workunit.client.0.smithi146.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T15:29:51.980 INFO:tasks.workunit.client.0.smithi146.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T15:29:51.980 INFO:tasks.workunit.client.0.smithi146.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stdout:0000073 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T15:29:51.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:29:53.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.dpg8S8AmsU_the_val 2024-03-09T15:29:54.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:29:54.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-03-09T15:29:54.068 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T15:29:54.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-03-09T15:29:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:29:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:29:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-03-09T15:29:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T15:29:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:29:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:29:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:29:54.080 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:29:54.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:29:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:29:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:29:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:29:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:29:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-03-09T15:29:54.083 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:29:54.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:29:54.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:29:54.114 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:54.113+0000 7f000bc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:54.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:54.116+0000 7f000bc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:54.117 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:54.117+0000 7f000bc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:29:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:29:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:29:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:29:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:29:54.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:29:54.367 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:29:54.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:29:54.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:29:54.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:29:54.630 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:54.630+0000 7f000bc0f640 -1 Falling back to public interface 2024-03-09T15:29:54.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:29:55.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:29:55.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:29:55.715 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:29:55.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:29:55.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:29:55.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:29:56.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:29:56.828 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:56.828+0000 7f000bc0f640 -1 osd.0 30 log_to_monitors true 2024-03-09T15:29:57.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:29:57.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:29:57.007 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:29:57.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:29:57.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:29:57.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:29:57.393 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 33 up_thru 18 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/197221088,v1:127.0.0.1:6803/197221088] [v2:127.0.0.1:6804/197221088,v1:127.0.0.1:6805/197221088] exists,up ec990f73-4e8e-478d-b8d5-6a61e00d77b1 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:29:57.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:29:57.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:29:57.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:29:57.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:29:57.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:29:57.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:29:57.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-03-09T15:29:57.399 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:29:57.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:29:57.432 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:57.432+0000 7fb18700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:57.449 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:57.449+0000 7fb18700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:57.450 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:57.450+0000 7fb18700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:29:57.775 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:29:57.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:29:57.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:29:57.964 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:29:57.964+0000 7fb18700d640 -1 Falling back to public interface 2024-03-09T15:29:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:29:59.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:29:59.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:29:59.083 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:29:59.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:29:59.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:29:59.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:29:59.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:00.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:00.221+0000 7fb18700d640 -1 osd.1 30 log_to_monitors true 2024-03-09T15:30:00.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:00.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:00.402 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:00.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:30:00.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:00.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:30:00.720 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 36 up_thru 18 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/960903946,v1:127.0.0.1:6811/960903946] [v2:127.0.0.1:6812/960903946,v1:127.0.0.1:6813/960903946] exists,up 4c99ae61-9a84-42d0-8d8f-5138063a1025 2024-03-09T15:30:00.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:30:00.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:30:00.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:30:00.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-03-09T15:30:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T15:30:00.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T15:30:00.727 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fa69db20-4d7d-4965-82fc-b507710c3877 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28442/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:30:00.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:30:00.760 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:00.760+0000 7f1d3f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:00.776 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:00.776+0000 7f1d3f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:00.777 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:00.777+0000 7f1d3f20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:30:01.082 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:30:01.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:01.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:01.299 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:01.299+0000 7f1d3f20d640 -1 Falling back to public interface 2024-03-09T15:30:01.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:02.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:02.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:02.421 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:30:02.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:30:02.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:02.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:02.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:03.609 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:03.609+0000 7f1d3f20d640 -1 osd.2 30 log_to_monitors true 2024-03-09T15:30:03.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:03.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:03.713 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:03.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:30:03.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:03.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:04.074 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 40 up_thru 17 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/2008561897,v1:127.0.0.1:6819/2008561897] [v2:127.0.0.1:6820/2008561897,v1:127.0.0.1:6821/2008561897] exists,up 14001c63-1fe1-4ae0-a4ce-6d8b246a05ec 2024-03-09T15:30:04.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:30:04.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:30:04.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:30:04.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-03-09T15:30:05.076 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T15:30:05.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-03-09T15:30:05.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:30:05.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-03-09T15:30:05.184 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:05.184 INFO:tasks.workunit.client.0.smithi146.stdout: "version": "19.0.0-1817-ga96b1c04", 2024-03-09T15:30:05.184 INFO:tasks.workunit.client.0.smithi146.stdout: "release": "squid", 2024-03-09T15:30:05.184 INFO:tasks.workunit.client.0.smithi146.stdout: "release_type": "dev" 2024-03-09T15:30:05.184 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:05.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:30:05.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-03-09T15:30:05.307 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:05.307 INFO:tasks.workunit.client.0.smithi146.stdout: "version": "19.0.0-1817-ga96b1c04", 2024-03-09T15:30:05.307 INFO:tasks.workunit.client.0.smithi146.stdout: "release": "squid", 2024-03-09T15:30:05.307 INFO:tasks.workunit.client.0.smithi146.stdout: "release_type": "dev" 2024-03-09T15:30:05.307 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:05.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T15:30:05.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-03-09T15:30:05.450 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:05.450 INFO:tasks.workunit.client.0.smithi146.stdout: "version": "19.0.0-1817-ga96b1c04", 2024-03-09T15:30:05.450 INFO:tasks.workunit.client.0.smithi146.stdout: "release": "squid", 2024-03-09T15:30:05.450 INFO:tasks.workunit.client.0.smithi146.stdout: "release_type": "dev" 2024-03-09T15:30:05.450 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:05.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-09T15:30:05.486 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:05.487 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "objxxx", 2024-03-09T15:30:05.487 INFO:tasks.workunit.client.0.smithi146.stdout: "clones": [ 2024-03-09T15:30:05.487 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.487 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 2, 2024-03-09T15:30:05.487 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 1, 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap01" 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.488 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 2, 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap02" 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:30:05.489 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 3, 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 3, 2024-03-09T15:30:05.490 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap13" 2024-03-09T15:30:05.491 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:30:05.491 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:30:05.491 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:30:05.491 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:30:05.492 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:30:05.492 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.492 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 5, 2024-03-09T15:30:05.492 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [ 2024-03-09T15:30:05.492 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 4, 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap24" 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "id": 5, 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "name": "snap25" 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 6, 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "overlaps": [] 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: { 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "id": "head", 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "snapshots": [], 2024-03-09T15:30:05.493 INFO:tasks.workunit.client.0.smithi146.stdout: "size": 5 2024-03-09T15:30:05.494 INFO:tasks.workunit.client.0.smithi146.stdout: } 2024-03-09T15:30:05.494 INFO:tasks.workunit.client.0.smithi146.stdout: ] 2024-03-09T15:30:05.494 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:05.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-03-09T15:30:05.868 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is unset 2024-03-09T15:30:05.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-03-09T15:30:06.370 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is unset 2024-03-09T15:30:06.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T15:30:06.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-09T15:30:06.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-03-09T15:30:06.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T15:30:05.067897+0000 28'7 42:66 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-09T15:29:26.083712+0000 28'7 2024-03-09T15:29:26.083712+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:29:26.083712+0000 4 0 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-09T15:30:01.421889+0000 0'0 38:29 [0,1] 0 [0,1] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-09T15:30:01.421885+0000 0'0 38:18 [0,1] 0 [0,1] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:05.067675+0000 0'0 42:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:30:06.924 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:30:06.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-03-09T15:30:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T15:30:09.042 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:09.042 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:30:09.042 INFO:tasks.workunit.client.0.smithi146.stdout: "must": true, 2024-03-09T15:30:09.042 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "0.000000" 2024-03-09T15:30:09.042 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:09.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-03-09T15:30:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T15:30:09.244359+0000 28'7 44:80 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-09T15:30:09.244273+0000 28'7 2024-03-09T15:30:09.244273+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:30:09.244273+0000 4 0 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:04.614450+0000 0'0 44:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:05.067620+0000 0'0 44:27 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:05.067675+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:14.315 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:30:14.316 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:30:14.316 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:30:14.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:30:14.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T15:30:14.327 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:30:09.243+0000 7fb16918f640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-09T15:30:14.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-03-09T15:30:14.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T15:30:14.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-03-09T15:30:14.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-03-09T15:30:14.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-03-09T15:30:14.331 INFO:tasks.workunit.client.0.smithi146.stdout:prev count: 1 2024-03-09T15:30:14.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T15:30:14.426 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:30:14.426 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:30:14.426 INFO:tasks.workunit.client.0.smithi146.stdout: "must": true, 2024-03-09T15:30:14.426 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "0.000000" 2024-03-09T15:30:14.426 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:30:14.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-03-09T15:30:19.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T15:30:19.700 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:30:19.700 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T15:30:15.206285+0000 28'7 44:89 [1,2,0] 1 [1,2,0] 1 28'7 2024-03-09T15:30:15.206192+0000 28'7 2024-03-09T15:30:15.206192+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:30:15.206192+0000 4 0 2024-03-09T15:30:19.700 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:04.614450+0000 0'0 44:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:19.700 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:05.067620+0000 0'0 44:27 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:19.701 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:30:05.067675+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:29:15.417677+0000 0'0 2024-03-09T15:29:15.417677+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:29:15.417677+0000 0 0 2024-03-09T15:30:19.701 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:30:19.701 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:30:19.701 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:30:19.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:30:19.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T15:30:19.712 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:30:09.243+0000 7fb16918f640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-09T15:30:19.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T15:30:19.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-03-09T15:30:19.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-03-09T15:30:19.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T15:30:19.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-03-09T15:30:19.715 INFO:tasks.workunit.client.0.smithi146.stdout:current count: 1 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:30:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:30:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:30:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-09T15:30:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T15:30:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T15:30:20.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-03-09T15:30:20.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:30:21.311 INFO:tasks.workunit.client.0.smithi146.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T15:30:21.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T15:30:21.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-09T15:30:21.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T15:30:21.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:30:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:30:22.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:30:22.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:30:22.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:30:22.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:30:22.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:30:22.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:30:22.207 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:30:22.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:30:22.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:30:22.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:30:22.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:30:22.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T15:30:22.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:30:22.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:30:22.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28442 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T15:30:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:30:22.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:30:22.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:30:22.229 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:30:22.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:30:22.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:30:22.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:30:22.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:30:22.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:30:22.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:30:22.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:30:22.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:30:22.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T15:30:22.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T15:30:22.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:30:22.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28442 2024-03-09T15:30:22.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28442 2024-03-09T15:30:22.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:30:22.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:30:22.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T15:30:22.237 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T15:30:22.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T15:30:22.307 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-03-09T15:30:22.307 DEBUG:teuthology.orchestra.run.smithi146:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-09T15:30:22.368 INFO:tasks.workunit.client.0.smithi146.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T15:30:22.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none ' 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T15:30:22.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-03-09T15:30:22.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T15:30:22.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-03-09T15:30:22.381 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_recovery_scrub_2' 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.44644 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-03-09T15:30:22.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T15:30:22.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:30:22.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:30:22.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:22.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44644 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:22.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44644 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:30:22.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:30:22.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:30:22.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-09T15:30:22.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:22.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:30:22.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:30:22.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:30:22.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:30:22.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44644/ceph-mon.a.asok 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:30:22.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44644/ceph-mon.a.asok config get fsid 2024-03-09T15:30:23.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:30:23.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:30:23.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:30:23.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:30:23.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44644/ceph-mon.a.asok 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:30:23.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44644/ceph-mon.a.asok config get mon_host 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-09T15:30:23.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:23.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:23.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:30:23.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:30:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-09T15:30:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-09T15:30:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-09T15:30:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-09T15:30:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T15:30:23.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:30:23.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:23.931 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:30:23.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5bbba46-8889-4c6f-96aa-a6a474f86844 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a5bbba46-8889-4c6f-96aa-a6a474f86844' 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 a5bbba46-8889-4c6f-96aa-a6a474f86844 2024-03-09T15:30:23.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:30:23.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPgOxlyCl+OBAApq3bfzslPa0pH4PM9jLYqw== 2024-03-09T15:30:23.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPgOxlyCl+OBAApq3bfzslPa0pH4PM9jLYqw=="}' 2024-03-09T15:30:23.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5bbba46-8889-4c6f-96aa-a6a474f86844 -i td/osd-recovery-scrub/0/new.json 2024-03-09T15:30:24.154 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:30:24.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-09T15:30:24.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCPgOxlyCl+OBAApq3bfzslPa0pH4PM9jLYqw== --osd-uuid a5bbba46-8889-4c6f-96aa-a6a474f86844 2024-03-09T15:30:24.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.193+0000 7fdb81e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:24.204 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.205+0000 7fdb81e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.206+0000 7fdb81e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.206+0000 7fdb81e68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:30:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.206+0000 7fdb81e68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:30:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.206+0000 7fdb81e68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:30:24.298 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:24.297+0000 7fdb81e68640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-09T15:30:29.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-09T15:30:29.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:30:29.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:30:29.779 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:30:29.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:30:30.133 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:30:30.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:30:30.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T15:30:30.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:30:30.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:30:30.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:30:30.160 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:30.160+0000 7f6b74614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:30.179 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:30.180+0000 7f6b74614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:30.180 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:30.181+0000 7f6b74614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:30.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:30:30.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:30:30.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:30.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:30:30.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:30.703 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:30.703+0000 7f6b74614640 -1 Falling back to public interface 2024-03-09T15:30:31.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:31.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:31.679 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:30:31.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:30:31.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:31.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:30:31.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:32.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:32.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:32.951 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:32.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:30:32.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:32.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:30:33.028 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:33.027+0000 7f6b74614640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:30:33.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:34.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:34.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:34.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:30:34.271 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:30:34.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:34.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:30:34.551 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/657379680,v1:127.0.0.1:6803/657379680] [v2:127.0.0.1:6804/657379680,v1:127.0.0.1:6805/657379680] exists,up a5bbba46-8889-4c6f-96aa-a6a474f86844 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-03-09T15:30:34.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:30:34.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:30:34.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5961717a-92b2-4a65-b402-8fe76e5dc887 2024-03-09T15:30:34.558 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 5961717a-92b2-4a65-b402-8fe76e5dc887 2024-03-09T15:30:34.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5961717a-92b2-4a65-b402-8fe76e5dc887' 2024-03-09T15:30:34.559 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:30:34.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCagOxl5tgQIhAAJTrBMkDBQnu9eMI+IGldlA== 2024-03-09T15:30:34.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCagOxl5tgQIhAAJTrBMkDBQnu9eMI+IGldlA=="}' 2024-03-09T15:30:34.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5961717a-92b2-4a65-b402-8fe76e5dc887 -i td/osd-recovery-scrub/1/new.json 2024-03-09T15:30:34.938 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:30:34.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-09T15:30:34.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCagOxl5tgQIhAAJTrBMkDBQnu9eMI+IGldlA== --osd-uuid 5961717a-92b2-4a65-b402-8fe76e5dc887 2024-03-09T15:30:34.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:34.978+0000 7f7b5ae0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:35.031 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.030+0000 7f7b5ae0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:35.032 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.031+0000 7f7b5ae0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:35.033 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.031+0000 7f7b5ae0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:30:35.033 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.031+0000 7f7b5ae0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:30:35.033 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.031+0000 7f7b5ae0c640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:30:35.106 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:35.105+0000 7f7b5ae0c640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-09T15:30:41.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-09T15:30:41.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:30:41.172 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:30:41.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:30:41.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:30:41.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:30:41.522 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:30:41.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T15:30:41.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:30:41.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:30:41.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:30:41.548 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:41.547+0000 7f51d3411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:41.550 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:41.549+0000 7f51d3411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:41.551 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:41.550+0000 7f51d3411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:41.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:30:41.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:30:41.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:30:41.797 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:30:41.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:41.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:30:42.072 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:42.071+0000 7f51d3411640 -1 Falling back to public interface 2024-03-09T15:30:42.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:43.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:43.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:43.107 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:30:43.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:30:43.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:43.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:30:43.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:44.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:44.390 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:44.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:44.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:30:44.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:44.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:30:44.519 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:44.518+0000 7f51d3411640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:30:44.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:45.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:45.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:45.729 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:30:45.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:30:45.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:45.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:30:46.086 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3148681054,v1:127.0.0.1:6811/3148681054] [v2:127.0.0.1:6812/3148681054,v1:127.0.0.1:6813/3148681054] exists,up 5961717a-92b2-4a65-b402-8fe76e5dc887 2024-03-09T15:30:46.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:30:46.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:30:46.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:30:46.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:30:46.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:30:46.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:30:46.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3842cc4d-dbf2-4caa-ad43-25654a769249 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 3842cc4d-dbf2-4caa-ad43-25654a769249 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3842cc4d-dbf2-4caa-ad43-25654a769249' 2024-03-09T15:30:46.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:30:46.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmgOxlX69rBhAAk07aCalz9652pCw8tFxyaQ== 2024-03-09T15:30:46.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmgOxlX69rBhAAk07aCalz9652pCw8tFxyaQ=="}' 2024-03-09T15:30:46.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3842cc4d-dbf2-4caa-ad43-25654a769249 -i td/osd-recovery-scrub/2/new.json 2024-03-09T15:30:46.463 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:46.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-09T15:30:46.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCmgOxlX69rBhAAk07aCalz9652pCw8tFxyaQ== --osd-uuid 3842cc4d-dbf2-4caa-ad43-25654a769249 2024-03-09T15:30:46.503 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.503+0000 7fdeeec0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:46.540 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.539+0000 7fdeeec0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:46.540 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.540+0000 7fdeeec0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.540+0000 7fdeeec0f640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:30:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.540+0000 7fdeeec0f640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:30:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.540+0000 7fdeeec0f640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:30:46.606 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:46.606+0000 7fdeeec0f640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-09T15:30:52.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-09T15:30:52.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:30:52.246 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:30:52.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:30:52.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:30:52.597 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:30:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:30:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T15:30:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:30:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:30:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:30:52.624 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:52.624+0000 7f8c75215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:52.626 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:52.626+0000 7f8c75215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:52.627 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:52.627+0000 7f8c75215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:30:52.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:52.868 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:30:52.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:30:52.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:52.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:53.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:53.148 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:53.148+0000 7f8c75215640 -1 Falling back to public interface 2024-03-09T15:30:54.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:54.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:54.142 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:30:54.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:30:54.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:54.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:54.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:55.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:55.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:55.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:30:55.415 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:30:55.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:55.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:55.462 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:55.462+0000 7f8c75215640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:30:55.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:30:56.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:30:56.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:30:56.763 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:30:56.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:30:56.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:30:56.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:30:57.038 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/340935775,v1:127.0.0.1:6819/340935775] [v2:127.0.0.1:6820/340935775,v1:127.0.0.1:6821/340935775] exists,up 3842cc4d-dbf2-4caa-ad43-25654a769249 2024-03-09T15:30:57.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:30:57.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:30:57.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:30:57.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:30:57.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:30:57.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:30:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd468951-5755-411c-a066-1450052445a4 2024-03-09T15:30:57.045 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 dd468951-5755-411c-a066-1450052445a4 2024-03-09T15:30:57.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dd468951-5755-411c-a066-1450052445a4' 2024-03-09T15:30:57.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:30:57.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxgOxlO5icAxAAfwBTf9T+tFDHkUsTVuAKWw== 2024-03-09T15:30:57.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxgOxlO5icAxAAfwBTf9T+tFDHkUsTVuAKWw=="}' 2024-03-09T15:30:57.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd468951-5755-411c-a066-1450052445a4 -i td/osd-recovery-scrub/3/new.json 2024-03-09T15:30:57.414 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:30:57.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-09T15:30:57.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCxgOxlO5icAxAAfwBTf9T+tFDHkUsTVuAKWw== --osd-uuid dd468951-5755-411c-a066-1450052445a4 2024-03-09T15:30:57.457 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.455+0000 7f9fbcc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:57.499 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.499+0000 7f9fbcc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:57.499 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.500+0000 7f9fbcc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:30:57.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.500+0000 7f9fbcc2a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:30:57.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.500+0000 7f9fbcc2a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:30:57.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.500+0000 7f9fbcc2a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:30:57.574 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:30:57.574+0000 7f9fbcc2a640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-09T15:31:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-09T15:31:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:31:03.382 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T15:31:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T15:31:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:31:03.737 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T15:31:03.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T15:31:03.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T15:31:03.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:31:03.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:31:03.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:31:03.763 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:31:03.763+0000 7fd1cfc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:31:03.765 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:31:03.765+0000 7fd1cfc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:31:03.766 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:31:03.766+0000 7fd1cfc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:31:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:31:04.058 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:31:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:31:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:31:04.296 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:31:04.296+0000 7fd1cfc14640 -1 Falling back to public interface 2024-03-09T15:31:04.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:31:05.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:31:05.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:31:05.352 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:31:05.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:31:05.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:31:05.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:31:05.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:31:06.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:31:06.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:31:06.636 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:31:06.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:31:06.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:31:06.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:31:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:31:06.787+0000 7fd1cfc14640 -1 osd.3 0 log_to_monitors true 2024-03-09T15:31:06.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:31:07.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:31:07.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:31:07.988 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:31:07.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:31:07.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:31:07.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:31:08.276 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1466609280,v1:127.0.0.1:6827/1466609280] [v2:127.0.0.1:6828/1466609280,v1:127.0.0.1:6829/1466609280] exists,up dd468951-5755-411c-a066-1450052445a4 2024-03-09T15:31:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:31:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:31:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:31:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-03-09T15:31:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-03-09T15:31:08.640 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'test' created 2024-03-09T15:31:08.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:31:09.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-03-09T15:31:09.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:31:09.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:31:09.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:31:09.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:31:09.654 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:31:09.654 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:31:09.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:31:09.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:31:09.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:31:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:31:10.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:31:10.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-03-09T15:31:10.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-03-09T15:31:10.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-03-09T15:31:10.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:31:10.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:31:10.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T15:31:10.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T15:31:10.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-03-09T15:31:10.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:31:10.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:31:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-09T15:31:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-09T15:31:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2024-03-09T15:31:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:31:10.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:31:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-03-09T15:31:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-03-09T15:31:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2024-03-09T15:31:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:31:10.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-03-09T15:31:10.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:31:10.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:31:10.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-03-09T15:31:10.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:31:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-03-09T15:31:10.477 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836489 2024-03-09T15:31:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-03-09T15:31:10.477 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:31:10.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-09T15:31:10.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:31:11.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:31:11.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:31:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-09T15:31:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:31:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:31:13.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:31:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-03-09T15:31:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:31:13.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T15:31:13.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:31:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:31:13.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T15:31:13.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:31:13.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T15:31:13.316 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672966 2024-03-09T15:31:13.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T15:31:13.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:31:13.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-09T15:31:13.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:31:13.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-09T15:31:13.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:31:13.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:31:13.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-09T15:31:13.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:31:13.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-09T15:31:13.590 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509444 2024-03-09T15:31:13.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-09T15:31:13.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:31:13.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-03-09T15:31:13.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:31:13.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-03-09T15:31:13.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:31:13.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:31:13.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-03-09T15:31:13.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:31:13.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-03-09T15:31:13.862 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345922 2024-03-09T15:31:13.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-03-09T15:31:13.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:31:14.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345922 2024-03-09T15:31:14.148 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:31:14.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:31:14.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:31:14.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:31:14.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:31:14.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:31:14.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:31:14.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:31:14.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:31:14.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:31:15.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:31:15.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:31:15.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:31:15.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-03-09T15:31:15.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-03-09T15:31:15.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-03-09T15:31:15.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-03-09T15:31:15.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-09T15:31:15.615 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:31:15.615 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:31:09.349145+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-03-09T15:31:08.594621+0000 0'0 2024-03-09T15:31:08.594621+0000 0 0 periodic scrub scheduled @ 2024-03-10T23:16:21.554397+0000 0 0 2024-03-09T15:31:15.615 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:31:15.615 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:31:15.616 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:31:15.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.44644 bs=1M count=50 2024-03-09T15:31:15.767 INFO:tasks.workunit.client.0.smithi146.stderr:50+0 records in 2024-03-09T15:31:15.767 INFO:tasks.workunit.client.0.smithi146.stderr:50+0 records out 2024-03-09T15:31:15.767 INFO:tasks.workunit.client.0.smithi146.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.140218 s, 374 MB/s 2024-03-09T15:31:15.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-03-09T15:31:15.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:15.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.44644 2024-03-09T15:31:17.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:17.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.44644 2024-03-09T15:31:18.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:18.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.44644 2024-03-09T15:31:20.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:20.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.44644 2024-03-09T15:31:21.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:21.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.44644 2024-03-09T15:31:23.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:23.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.44644 2024-03-09T15:31:24.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:24.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.44644 2024-03-09T15:31:25.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:25.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.44644 2024-03-09T15:31:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.44644 2024-03-09T15:31:28.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:28.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.44644 2024-03-09T15:31:30.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:30.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.44644 2024-03-09T15:31:31.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:31.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.44644 2024-03-09T15:31:33.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:33.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.44644 2024-03-09T15:31:34.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:34.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.44644 2024-03-09T15:31:36.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:36.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.44644 2024-03-09T15:31:37.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:37.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.44644 2024-03-09T15:31:39.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:39.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.44644 2024-03-09T15:31:40.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:40.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.44644 2024-03-09T15:31:42.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:42.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.44644 2024-03-09T15:31:43.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:43.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.44644 2024-03-09T15:31:45.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:45.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.44644 2024-03-09T15:31:46.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:46.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.44644 2024-03-09T15:31:48.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:48.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.44644 2024-03-09T15:31:49.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:49.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.44644 2024-03-09T15:31:51.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:51.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.44644 2024-03-09T15:31:52.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:52.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.44644 2024-03-09T15:31:54.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:54.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.44644 2024-03-09T15:31:55.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:55.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.44644 2024-03-09T15:31:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.44644 2024-03-09T15:31:58.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:58.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.44644 2024-03-09T15:31:59.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:31:59.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.44644 2024-03-09T15:32:01.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:01.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.44644 2024-03-09T15:32:02.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:02.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.44644 2024-03-09T15:32:04.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:04.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.44644 2024-03-09T15:32:05.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:05.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.44644 2024-03-09T15:32:07.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:07.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.44644 2024-03-09T15:32:08.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:08.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.44644 2024-03-09T15:32:10.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:10.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.44644 2024-03-09T15:32:11.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:11.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.44644 2024-03-09T15:32:13.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:13.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.44644 2024-03-09T15:32:14.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:14.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.44644 2024-03-09T15:32:16.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:16.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.44644 2024-03-09T15:32:17.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:17.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.44644 2024-03-09T15:32:19.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:19.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.44644 2024-03-09T15:32:20.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:20.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.44644 2024-03-09T15:32:22.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:22.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.44644 2024-03-09T15:32:23.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:23.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.44644 2024-03-09T15:32:25.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:25.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.44644 2024-03-09T15:32:26.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:26.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.44644 2024-03-09T15:32:28.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:28.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.44644 2024-03-09T15:32:29.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:29.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.44644 2024-03-09T15:32:31.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:31.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.44644 2024-03-09T15:32:32.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:32.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.44644 2024-03-09T15:32:34.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:34.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.44644 2024-03-09T15:32:35.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:35.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.44644 2024-03-09T15:32:37.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:37.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.44644 2024-03-09T15:32:38.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:38.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.44644 2024-03-09T15:32:40.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:40.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.44644 2024-03-09T15:32:41.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:41.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.44644 2024-03-09T15:32:43.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:43.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.44644 2024-03-09T15:32:44.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:44.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.44644 2024-03-09T15:32:46.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:46.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.44644 2024-03-09T15:32:47.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:47.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.44644 2024-03-09T15:32:48.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:48.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.44644 2024-03-09T15:32:50.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:50.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.44644 2024-03-09T15:32:51.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:51.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.44644 2024-03-09T15:32:53.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:53.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.44644 2024-03-09T15:32:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:54.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.44644 2024-03-09T15:32:56.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:56.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.44644 2024-03-09T15:32:57.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:57.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.44644 2024-03-09T15:32:59.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:32:59.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.44644 2024-03-09T15:33:00.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:00.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.44644 2024-03-09T15:33:01.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:01.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.44644 2024-03-09T15:33:03.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:03.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.44644 2024-03-09T15:33:04.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:04.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.44644 2024-03-09T15:33:06.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:06.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.44644 2024-03-09T15:33:07.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:07.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.44644 2024-03-09T15:33:09.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:09.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.44644 2024-03-09T15:33:10.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:10.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.44644 2024-03-09T15:33:12.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:12.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.44644 2024-03-09T15:33:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.44644 2024-03-09T15:33:15.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:15.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.44644 2024-03-09T15:33:16.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:16.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.44644 2024-03-09T15:33:18.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:18.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.44644 2024-03-09T15:33:19.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:19.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.44644 2024-03-09T15:33:21.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:21.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.44644 2024-03-09T15:33:22.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:22.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.44644 2024-03-09T15:33:24.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:24.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.44644 2024-03-09T15:33:25.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:25.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.44644 2024-03-09T15:33:27.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:27.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.44644 2024-03-09T15:33:28.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:28.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.44644 2024-03-09T15:33:29.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:29.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.44644 2024-03-09T15:33:31.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:31.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.44644 2024-03-09T15:33:32.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:32.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.44644 2024-03-09T15:33:34.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:34.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.44644 2024-03-09T15:33:35.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:35.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.44644 2024-03-09T15:33:37.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:37.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.44644 2024-03-09T15:33:38.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:38.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.44644 2024-03-09T15:33:40.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:40.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.44644 2024-03-09T15:33:41.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T15:33:41.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.44644 2024-03-09T15:33:43.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.44644 2024-03-09T15:33:43.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-03-09T15:33:43.831 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 4 2024-03-09T15:33:43.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-03-09T15:33:43.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-03-09T15:33:43.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T15:33:43.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T15:33:43.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T15:33:43.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T15:33:44.099 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:33:44.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T15:33:46.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-03-09T15:33:46.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-03-09T15:33:46.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-03-09T15:33:46.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T15:33:46.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T15:33:46.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T15:33:46.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T15:33:46.408 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:33:46.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T15:33:48.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-03-09T15:33:48.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-03-09T15:33:48.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-03-09T15:33:48.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T15:33:48.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T15:33:48.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T15:33:48.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T15:33:48.691 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:33:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-03-09T15:33:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-03-09T15:33:48.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-09T15:33:48.963 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:33:48.964 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-03-09T15:33:45.326722+0000 24'1300 27:1323 [1,0,2,3] 1 [1,0] 1 0'0 2024-03-09T15:31:08.594621+0000 0'0 2024-03-09T15:31:08.594621+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:33:48.964 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:33:48.964 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:33:48.964 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:33:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-03-09T15:33:58.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-03-09T15:33:58.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:33:58.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:33:58.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:33:58.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:33:58.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:34:04.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:34:04.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-03-09T15:34:04.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-03-09T15:34:04.291 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T15:34:04.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-03-09T15:34:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:50.022+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:52.019+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:53.013+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:56.112+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:58.152+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.296 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:00.126+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.297 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:01.123+0000 7f6b6cf6e640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:51.841+0000 7f51cbdbc640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:52.867+0000 7f51cbdbc640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:53.842+0000 7f51cbdbc640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:56.858+0000 7f51cbdbc640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:58.878+0000 7f51cbdbc640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:57.755+0000 7f8c6db92640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:58.773+0000 7f8c6db92640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:00.833+0000 7f8c6db92640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:01.849+0000 7f8c6db92640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:03.867+0000 7f8c6db92640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-09T15:34:04.314 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:49.264+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:50.280+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:53.301+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:55.252+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:56.217+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:33:58.217+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.315 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:34:03.357+0000 7fd1c85f9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T15:34:04.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-03-09T15:34:04.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-03-09T15:34:04.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-03-09T15:34:04.316 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T15:34:04.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-03-09T15:34:04.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-09T15:34:04.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T15:34:04.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-03-09T15:34:04.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-03-09T15:34:04.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-09T15:34:04.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T15:34:04.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-03-09T15:34:04.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-03-09T15:34:04.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-09T15:34:04.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T15:34:04.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-03-09T15:34:04.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-03-09T15:34:04.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:04.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-09T15:34:04.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T15:34:04.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-03-09T15:34:04.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-03-09T15:34:04.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-03-09T15:34:04.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:34:04.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:34:04.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:34:04.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:34:04.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:34:04.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:34:04.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:34:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:34:04.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:34:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:34:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:34:04.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:34:04.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:34:04.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:34:04.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:34:04.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T15:34:04.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:34:04.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:04.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:04.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44644 2024-03-09T15:34:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:34:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:34:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-03-09T15:34:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-03-09T15:34:04.377 INFO:tasks.workunit.client.0.smithi146.stdout:TEST PASSED 2024-03-09T15:34:04.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.44644 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-09T15:34:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:34:04.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:34:04.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:34:04.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:34:04.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:34:04.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:34:04.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:34:04.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:34:04.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:34:04.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:34:04.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:34:04.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:34:04.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:34:04.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T15:34:04.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:34:04.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:04.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:04.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44644 2024-03-09T15:34:04.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:34:04.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:34:04.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-09T15:34:04.388 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:34:04.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:04.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:04.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44644 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:34:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-09T15:34:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:34:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:34:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:34:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-09T15:34:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:04.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T15:34:04.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:34:04.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:34:04.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:34:04.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:34:04.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44644/ceph-mon.a.asok 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:34:04.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44644/ceph-mon.a.asok config get fsid 2024-03-09T15:34:05.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:34:05.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:34:05.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:34:05.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44644/ceph-mon.a.asok 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:34:05.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44644/ceph-mon.a.asok config get mon_host 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-09T15:34:05.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:34:05.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:34:05.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:05.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:34:05.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:34:05.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-09T15:34:05.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-09T15:34:05.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-09T15:34:05.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-09T15:34:05.950 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T15:34:05.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:34:05.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:34:05.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:34:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9816e9ef-7eef-4be6-8890-75423b9f37fc 2024-03-09T15:34:05.958 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 9816e9ef-7eef-4be6-8890-75423b9f37fc 2024-03-09T15:34:05.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9816e9ef-7eef-4be6-8890-75423b9f37fc' 2024-03-09T15:34:05.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:34:05.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtgexlgXf8ORAAa3I3DNGCUkiLWMITX3D5jQ== 2024-03-09T15:34:05.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtgexlgXf8ORAAa3I3DNGCUkiLWMITX3D5jQ=="}' 2024-03-09T15:34:05.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9816e9ef-7eef-4be6-8890-75423b9f37fc -i td/osd-recovery-scrub/0/new.json 2024-03-09T15:34:06.178 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-09T15:34:06.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBtgexlgXf8ORAAa3I3DNGCUkiLWMITX3D5jQ== --osd-uuid 9816e9ef-7eef-4be6-8890-75423b9f37fc 2024-03-09T15:34:06.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.220+0000 7f62a181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:06.229 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.228+0000 7f62a181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:06.230 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.229+0000 7f62a181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:06.230 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.229+0000 7f62a181e640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:34:06.230 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.230+0000 7f62a181e640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:34:06.231 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.230+0000 7f62a181e640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T15:34:06.321 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:06.320+0000 7f62a181e640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-09T15:34:11.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-09T15:34:11.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:34:11.844 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:34:11.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:34:11.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:34:12.231 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:34:12.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:34:12.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:12.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:34:12.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:34:12.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:34:12.261 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:12.260+0000 7f24bbbac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:12.300 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:12.300+0000 7f24bbbac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:12.301 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:12.301+0000 7f24bbbac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:12.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:34:12.504 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:12.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:12.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:34:12.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:12.831 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:12.831+0000 7f24bbbac640 -1 Falling back to public interface 2024-03-09T15:34:13.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:13.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:13.777 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:13.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:34:13.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:13.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:34:14.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:15.054 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:34:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:34:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:34:15.166 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:15.166+0000 7f24bbbac640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:34:15.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:16.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:16.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:16.351 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:34:16.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:34:16.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:16.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:34:16.680 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/442689732,v1:127.0.0.1:6803/442689732] [v2:127.0.0.1:6804/442689732,v1:127.0.0.1:6805/442689732] exists,up 9816e9ef-7eef-4be6-8890-75423b9f37fc 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:34:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:16.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:34:16.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=437cc2d3-3f38-424c-93fc-edc14573bf96 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 437cc2d3-3f38-424c-93fc-edc14573bf96' 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 437cc2d3-3f38-424c-93fc-edc14573bf96 2024-03-09T15:34:16.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:34:16.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4gexlCrjNKRAA2zyFghrk4VFBYe7CZ4PpIQ== 2024-03-09T15:34:16.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4gexlCrjNKRAA2zyFghrk4VFBYe7CZ4PpIQ=="}' 2024-03-09T15:34:16.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 437cc2d3-3f38-424c-93fc-edc14573bf96 -i td/osd-recovery-scrub/1/new.json 2024-03-09T15:34:17.035 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:17.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-09T15:34:17.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB4gexlCrjNKRAA2zyFghrk4VFBYe7CZ4PpIQ== --osd-uuid 437cc2d3-3f38-424c-93fc-edc14573bf96 2024-03-09T15:34:17.078 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.078+0000 7fcf8400d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:17.119 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.119+0000 7fcf8400d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:17.121 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.120+0000 7fcf8400d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:17.121 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.120+0000 7fcf8400d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:34:17.121 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.120+0000 7fcf8400d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:34:17.121 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.120+0000 7fcf8400d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T15:34:17.194 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:17.193+0000 7fcf8400d640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-09T15:34:23.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-09T15:34:23.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:34:23.018 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:34:23.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:34:23.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:34:23.371 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:34:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:34:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:34:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:34:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:34:23.401 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:23.402+0000 7fe992dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:23.403+0000 7fe992dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:23.405+0000 7fe992dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:23.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:34:23.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:34:23.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:23.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:23.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:34:23.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:23.928 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:23.929+0000 7fe992dbd640 -1 Falling back to public interface 2024-03-09T15:34:24.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:24.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:24.913 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:24.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:34:24.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:24.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:34:25.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:26.196 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:34:26.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:34:26.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:26.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:34:26.400 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:26.400+0000 7fe992dbd640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:34:26.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:27.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:27.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:27.501 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:34:27.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:34:27.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:27.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:34:27.783 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4233908205,v1:127.0.0.1:6811/4233908205] [v2:127.0.0.1:6812/4233908205,v1:127.0.0.1:6813/4233908205] exists,up 437cc2d3-3f38-424c-93fc-edc14573bf96 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:34:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:34:27.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:34:27.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ac2b37e-43ce-4ffd-ba53-b68e51c479ac 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4ac2b37e-43ce-4ffd-ba53-b68e51c479ac' 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 4ac2b37e-43ce-4ffd-ba53-b68e51c479ac 2024-03-09T15:34:27.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:34:27.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDgexlsFTuLxAA8MIiXOKAScYBj9sHMw7hHg== 2024-03-09T15:34:27.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDgexlsFTuLxAA8MIiXOKAScYBj9sHMw7hHg=="}' 2024-03-09T15:34:27.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ac2b37e-43ce-4ffd-ba53-b68e51c479ac -i td/osd-recovery-scrub/2/new.json 2024-03-09T15:34:28.169 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:34:28.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-09T15:34:28.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCDgexlsFTuLxAA8MIiXOKAScYBj9sHMw7hHg== --osd-uuid 4ac2b37e-43ce-4ffd-ba53-b68e51c479ac 2024-03-09T15:34:28.214 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.215+0000 7feeb639a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:28.254 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.254+0000 7feeb639a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:28.255 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.255+0000 7feeb639a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:28.255 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.256+0000 7feeb639a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:34:28.255 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.256+0000 7feeb639a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:34:28.255 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.256+0000 7feeb639a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T15:34:28.320 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:28.321+0000 7feeb639a640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-09T15:34:34.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-09T15:34:34.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:34:34.276 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:34:34.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:34:34.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:34:34.630 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:34:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:34:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:34:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:34:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:34:34.660 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:34.660+0000 7f8ea9019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:34.662 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:34.662+0000 7f8ea9019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:34.663 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:34.663+0000 7f8ea9019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:34:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:34.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:34:34.906 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:34.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:34.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:34:35.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:35.195 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:35.195+0000 7f8ea9019640 -1 Falling back to public interface 2024-03-09T15:34:36.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:36.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:36.175 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:34:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:34:36.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:37.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:37.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:37.456 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:34:37.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:34:37.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:37.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:34:37.626 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:37.626+0000 7f8ea9019640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:34:37.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:38.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:38.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:38.743 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:34:38.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:34:38.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:38.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:34:39.041 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4248027931,v1:127.0.0.1:6819/4248027931] [v2:127.0.0.1:6820/4248027931,v1:127.0.0.1:6821/4248027931] exists,up 4ac2b37e-43ce-4ffd-ba53-b68e51c479ac 2024-03-09T15:34:39.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:34:39.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:34:39.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:34:39.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:34:39.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=480c38be-e6c3-449f-ab69-c839947c5c4e 2024-03-09T15:34:39.049 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 480c38be-e6c3-449f-ab69-c839947c5c4e 2024-03-09T15:34:39.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 480c38be-e6c3-449f-ab69-c839947c5c4e' 2024-03-09T15:34:39.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:34:39.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPgexlHVG+AxAAPdkJq4wcGjxkt83DyM9Xsw== 2024-03-09T15:34:39.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPgexlHVG+AxAAPdkJq4wcGjxkt83DyM9Xsw=="}' 2024-03-09T15:34:39.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 480c38be-e6c3-449f-ab69-c839947c5c4e -i td/osd-recovery-scrub/3/new.json 2024-03-09T15:34:39.428 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:34:39.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-09T15:34:39.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCPgexlHVG+AxAAPdkJq4wcGjxkt83DyM9Xsw== --osd-uuid 480c38be-e6c3-449f-ab69-c839947c5c4e 2024-03-09T15:34:39.472 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.472+0000 7f0fc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:39.513 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.513+0000 7f0fc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:39.513 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.514+0000 7f0fc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.514+0000 7f0fc841e640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:34:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.514+0000 7f0fc841e640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:34:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.514+0000 7f0fc841e640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T15:34:39.594 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:39.594+0000 7f0fc841e640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-09T15:34:45.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-09T15:34:45.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:34:45.592 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T15:34:45.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T15:34:45.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:34:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T15:34:45.943 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T15:34:45.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:45.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:34:45.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:34:45.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:34:45.973 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:45.973+0000 7f02cb01c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:45.974 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:45.975+0000 7f02cb01c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:45.976 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:45.976+0000 7f02cb01c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:46.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T15:34:46.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:34:46.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:34:46.504 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:46.505+0000 7f02cb01c640 -1 Falling back to public interface 2024-03-09T15:34:46.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:47.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:47.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:47.527 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:47.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:34:47.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:47.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:34:47.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:48.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:48.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:48.809 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:34:48.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:34:48.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:48.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:34:48.842 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:48.843+0000 7f02cb01c640 -1 osd.3 0 log_to_monitors true 2024-03-09T15:34:49.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:50.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:50.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:50.177 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:34:50.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:34:50.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:50.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:34:50.454 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3927433023,v1:127.0.0.1:6827/3927433023] [v2:127.0.0.1:6828/3927433023,v1:127.0.0.1:6829/3927433023] exists,up 480c38be-e6c3-449f-ab69-c839947c5c4e 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:50.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:34:50.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:34:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=60d607b6-cbe3-4c7c-9dfb-3e801a163ebf 2024-03-09T15:34:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 60d607b6-cbe3-4c7c-9dfb-3e801a163ebf' 2024-03-09T15:34:50.462 INFO:tasks.workunit.client.0.smithi146.stdout:add osd4 60d607b6-cbe3-4c7c-9dfb-3e801a163ebf 2024-03-09T15:34:50.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:34:50.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCagexl1ptkHBAAjoEXbYDQzBAeswa2Ut/hPw== 2024-03-09T15:34:50.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCagexl1ptkHBAAjoEXbYDQzBAeswa2Ut/hPw=="}' 2024-03-09T15:34:50.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 60d607b6-cbe3-4c7c-9dfb-3e801a163ebf -i td/osd-recovery-scrub/4/new.json 2024-03-09T15:34:50.836 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:34:50.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-03-09T15:34:50.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCagexl1ptkHBAAjoEXbYDQzBAeswa2Ut/hPw== --osd-uuid 60d607b6-cbe3-4c7c-9dfb-3e801a163ebf 2024-03-09T15:34:50.880 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.880+0000 7f5e4020e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:50.921 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.922+0000 7f5e4020e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:50.922 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.923+0000 7f5e4020e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:50.923 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.923+0000 7f5e4020e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T15:34:50.923 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.923+0000 7f5e4020e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T15:34:50.923 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.923+0000 7f5e4020e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T15:34:50.988 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:50.988+0000 7f5e4020e640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-03-09T15:34:57.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-03-09T15:34:57.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:34:57.062 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd4 key to auth repository 2024-03-09T15:34:57.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T15:34:57.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:34:57.414 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.4 2024-03-09T15:34:57.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T15:34:57.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:34:57.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:34:57.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:34:57.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:34:57.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:57.445+0000 7fba32411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:57.446 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:57.447+0000 7fba32411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:57.447 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:57.448+0000 7fba32411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:34:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T15:34:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:34:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T15:34:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:34:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:34:57.688 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:34:57.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:57.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:34:57.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:57.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T15:34:57.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:34:57.971 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:34:57.971+0000 7fba32411640 -1 Falling back to public interface 2024-03-09T15:34:58.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:34:58.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:34:58.960 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:34:58.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:34:58.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:34:58.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T15:34:59.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:00.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:00.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:00.240 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:35:00.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:35:00.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:00.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T15:35:00.443 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:00.443+0000 7fba32411640 -1 osd.4 0 log_to_monitors true 2024-03-09T15:35:00.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:01.544 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:35:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:35:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T15:35:01.826 INFO:tasks.workunit.client.0.smithi146.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4084930569,v1:127.0.0.1:6835/4084930569] [v2:127.0.0.1:6836/4084930569,v1:127.0.0.1:6837/4084930569] exists,up 60d607b6-cbe3-4c7c-9dfb-3e801a163ebf 2024-03-09T15:35:01.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:35:01.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:35:01.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:35:01.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:35:01.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:35:01.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:35:01.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba7292e3-41b2-4d52-bfad-0814cb66043f 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ba7292e3-41b2-4d52-bfad-0814cb66043f' 2024-03-09T15:35:01.833 INFO:tasks.workunit.client.0.smithi146.stdout:add osd5 ba7292e3-41b2-4d52-bfad-0814cb66043f 2024-03-09T15:35:01.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:35:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClgexlpeSCMhAAOyHWRP/cWJ3sZtpRad9ZGw== 2024-03-09T15:35:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClgexlpeSCMhAAOyHWRP/cWJ3sZtpRad9ZGw=="}' 2024-03-09T15:35:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba7292e3-41b2-4d52-bfad-0814cb66043f -i td/osd-recovery-scrub/5/new.json 2024-03-09T15:35:02.280 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:35:02.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-03-09T15:35:02.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQClgexlpeSCMhAAOyHWRP/cWJ3sZtpRad9ZGw== --osd-uuid ba7292e3-41b2-4d52-bfad-0814cb66043f 2024-03-09T15:35:02.325 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.326+0000 7fdae71e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:02.365 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.365+0000 7fdae71e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.366+0000 7fdae71e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.367+0000 7fdae71e1640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T15:35:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.367+0000 7fdae71e1640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T15:35:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.367+0000 7fdae71e1640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T15:35:02.440 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:02.440+0000 7fdae71e1640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-03-09T15:35:08.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-03-09T15:35:08.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:35:08.390 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd5 key to auth repository 2024-03-09T15:35:08.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T15:35:08.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:35:08.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T15:35:08.748 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.5 2024-03-09T15:35:08.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:08.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:35:08.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:35:08.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:35:08.778 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:08.778+0000 7f7aaa21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:08.780 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:08.780+0000 7f7aaa21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:08.781 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:08.781+0000 7f7aaa21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:09.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T15:35:09.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:35:09.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T15:35:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:35:09.023 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:35:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:35:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:35:09.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:09.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T15:35:09.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:09.300 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:09.301+0000 7f7aaa21e640 -1 Falling back to public interface 2024-03-09T15:35:10.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:10.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:10.294 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:35:10.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:35:10.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:10.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T15:35:10.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:11.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:11.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:11.577 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:35:11.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:35:11.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:11.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T15:35:11.638 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:11.638+0000 7f7aaa21e640 -1 osd.5 0 log_to_monitors true 2024-03-09T15:35:11.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:12.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:12.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:12.872 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:35:12.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:35:12.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:12.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T15:35:13.153 INFO:tasks.workunit.client.0.smithi146.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1697233676,v1:127.0.0.1:6843/1697233676] [v2:127.0.0.1:6844/1697233676,v1:127.0.0.1:6845/1697233676] exists,up ba7292e3-41b2-4d52-bfad-0814cb66043f 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T15:35:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:35:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:35:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eced4b53-c169-479b-a39b-1e2be1fd0651 2024-03-09T15:35:13.161 INFO:tasks.workunit.client.0.smithi146.stdout:add osd6 eced4b53-c169-479b-a39b-1e2be1fd0651 2024-03-09T15:35:13.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 eced4b53-c169-479b-a39b-1e2be1fd0651' 2024-03-09T15:35:13.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:35:13.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxgexlDaBnChAAV4XqoNkQP9D34X3x0NAHeg== 2024-03-09T15:35:13.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxgexlDaBnChAAV4XqoNkQP9D34X3x0NAHeg=="}' 2024-03-09T15:35:13.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eced4b53-c169-479b-a39b-1e2be1fd0651 -i td/osd-recovery-scrub/6/new.json 2024-03-09T15:35:13.660 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:35:13.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-03-09T15:35:13.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCxgexlDaBnChAAV4XqoNkQP9D34X3x0NAHeg== --osd-uuid eced4b53-c169-479b-a39b-1e2be1fd0651 2024-03-09T15:35:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.703+0000 7fba6a571640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:13.768 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.767+0000 7fba6a571640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:13.769 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.769+0000 7fba6a571640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:13.769 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.769+0000 7fba6a571640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T15:35:13.770 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.769+0000 7fba6a571640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T15:35:13.770 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.769+0000 7fba6a571640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T15:35:13.861 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:13.859+0000 7fba6a571640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-03-09T15:35:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-03-09T15:35:20.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:35:20.022 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd6 key to auth repository 2024-03-09T15:35:20.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T15:35:20.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:35:20.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T15:35:20.404 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.6 2024-03-09T15:35:20.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:20.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:35:20.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:35:20.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:35:20.434 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:20.433+0000 7f107eddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:20.436 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:20.435+0000 7f107eddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:20.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:20.436+0000 7f107eddd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T15:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:35:20.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:20.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T15:35:20.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:20.968 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:20.968+0000 7f107eddd640 -1 Falling back to public interface 2024-03-09T15:35:21.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:21.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:21.955 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:35:21.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:35:21.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:21.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T15:35:22.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:23.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:23.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:23.234 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:35:23.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:35:23.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:23.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T15:35:23.312 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:23.311+0000 7f107eddd640 -1 osd.6 0 log_to_monitors true 2024-03-09T15:35:23.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:24.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:24.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:24.518 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:35:24.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:35:24.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:24.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T15:35:24.799 INFO:tasks.workunit.client.0.smithi146.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4011712706,v1:127.0.0.1:6851/4011712706] [v2:127.0.0.1:6852/4011712706,v1:127.0.0.1:6853/4011712706] exists,up eced4b53-c169-479b-a39b-1e2be1fd0651 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:35:24.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:35:24.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:35:24.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-03-09T15:35:24.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:35:24.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a90e94f3-098b-4d29-850c-e77a70ca56ff 2024-03-09T15:35:24.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 a90e94f3-098b-4d29-850c-e77a70ca56ff' 2024-03-09T15:35:24.808 INFO:tasks.workunit.client.0.smithi146.stdout:add osd7 a90e94f3-098b-4d29-850c-e77a70ca56ff 2024-03-09T15:35:24.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:35:24.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8gexlkKrhMBAATXRNvEAHi97OGY/Dq/ML+g== 2024-03-09T15:35:24.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8gexlkKrhMBAATXRNvEAHi97OGY/Dq/ML+g=="}' 2024-03-09T15:35:24.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a90e94f3-098b-4d29-850c-e77a70ca56ff -i td/osd-recovery-scrub/7/new.json 2024-03-09T15:35:25.191 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:35:25.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-03-09T15:35:25.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC8gexlkKrhMBAATXRNvEAHi97OGY/Dq/ML+g== --osd-uuid a90e94f3-098b-4d29-850c-e77a70ca56ff 2024-03-09T15:35:25.235 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.234+0000 7fa3b1a21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:25.267 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.267+0000 7fa3b1a21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.268+0000 7fa3b1a21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:25.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.268+0000 7fa3b1a21640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T15:35:25.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.268+0000 7fa3b1a21640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T15:35:25.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.268+0000 7fa3b1a21640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T15:35:25.342 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:25.342+0000 7fa3b1a21640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-03-09T15:35:31.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-03-09T15:35:31.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:35:31.543 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd7 key to auth repository 2024-03-09T15:35:31.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T15:35:31.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:35:31.895 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.7 2024-03-09T15:35:31.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T15:35:31.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=ec10ad5a-7c62-4396-b023-a3ec7749ecbc --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44644/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T15:35:31.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:35:31.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:35:31.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:35:31.927 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:31.927+0000 7fed5760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:31.928 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:31.929+0000 7fed5760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:31.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:31.930+0000 7fed5760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:35:32.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T15:35:32.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:35:32.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T15:35:32.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:35:32.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:35:32.190 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:35:32.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:32.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:35:32.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:32.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T15:35:32.459 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:32.460+0000 7fed5760d640 -1 Falling back to public interface 2024-03-09T15:35:32.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:33.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:33.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:33.465 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:35:33.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:35:33.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:33.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T15:35:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:34.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:34.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:34.747 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:35:34.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:35:34.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:34.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T15:35:34.786 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:35:34.786+0000 7fed5760d640 -1 osd.7 0 log_to_monitors true 2024-03-09T15:35:35.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:35:36.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:35:36.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:35:36.087 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:35:36.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:35:36.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:35:36.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T15:35:36.366 INFO:tasks.workunit.client.0.smithi146.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4171337195,v1:127.0.0.1:6859/4171337195] [v2:127.0.0.1:6860/4171337195,v1:127.0.0.1:6861/4171337195] exists,up a90e94f3-098b-4d29-850c-e77a70ca56ff 2024-03-09T15:35:36.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:35:36.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:35:36.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:35:36.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-03-09T15:35:36.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-03-09T15:35:36.757 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'test' created 2024-03-09T15:35:36.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:35:37.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:35:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:35:37.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:35:37.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:35:37.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:35:37.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:35:37.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:35:37.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:7' 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:35:38.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:38.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:35:38.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T15:35:38.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T15:35:38.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T15:35:38.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:38.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:35:38.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-09T15:35:38.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-09T15:35:38.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-03-09T15:35:38.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:38.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:35:39.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-09T15:35:39.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-09T15:35:39.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2024-03-09T15:35:39.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:39.144 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:35:39.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-03-09T15:35:39.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-03-09T15:35:39.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931' 2024-03-09T15:35:39.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:39.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T15:35:39.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-03-09T15:35:39.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-03-09T15:35:39.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409' 2024-03-09T15:35:39.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:39.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T15:35:39.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-09T15:35:39.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-09T15:35:39.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887' 2024-03-09T15:35:39.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:39.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T15:35:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-09T15:35:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-09T15:35:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2024-03-09T15:35:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:35:39.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T15:35:39.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:39.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:35:39.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T15:35:39.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:39.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T15:35:39.993 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836498 2024-03-09T15:35:39.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T15:35:39.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:35:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T15:35:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:40.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-09T15:35:40.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:40.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:35:40.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-09T15:35:40.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:40.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-09T15:35:40.316 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672976 2024-03-09T15:35:40.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-09T15:35:40.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:35:40.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-09T15:35:40.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:40.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-09T15:35:40.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:40.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:35:40.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-09T15:35:40.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:40.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-09T15:35:40.592 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509454 2024-03-09T15:35:40.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-09T15:35:40.592 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:35:40.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-09T15:35:40.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:40.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-03-09T15:35:40.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:40.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:35:40.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-03-09T15:35:40.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:40.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-03-09T15:35:40.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-03-09T15:35:40.876 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345931 2024-03-09T15:35:40.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:35:41.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-03-09T15:35:41.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:41.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-03-09T15:35:41.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:41.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T15:35:41.147 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-03-09T15:35:41.147 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:41.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-03-09T15:35:41.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-03-09T15:35:41.148 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.4 seq 107374182409 2024-03-09T15:35:41.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T15:35:41.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-03-09T15:35:41.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:41.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-03-09T15:35:41.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:41.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T15:35:41.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-03-09T15:35:41.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:41.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-09T15:35:41.425 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.5 seq 128849018887 2024-03-09T15:35:41.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-03-09T15:35:41.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T15:35:41.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-03-09T15:35:41.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:41.697 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-03-09T15:35:41.697 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:41.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T15:35:41.698 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-03-09T15:35:41.698 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:41.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-09T15:35:41.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-03-09T15:35:41.700 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.6 seq 150323855365 2024-03-09T15:35:41.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T15:35:41.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-09T15:35:41.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:35:41.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-03-09T15:35:41.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:35:41.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T15:35:41.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-03-09T15:35:41.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:35:41.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T15:35:41.986 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.7 seq 171798691842 2024-03-09T15:35:41.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-03-09T15:35:41.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T15:35:42.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-03-09T15:35:42.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:35:42.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:35:42.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:35:42.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:35:42.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-03-09T15:35:42.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:35:42.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:35:42.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-03-09T15:35:43.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-03-09T15:35:43.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-03-09T15:35:43.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.44644 bs=1M count=50 2024-03-09T15:35:43.629 INFO:tasks.workunit.client.0.smithi146.stderr:50+0 records in 2024-03-09T15:35:43.630 INFO:tasks.workunit.client.0.smithi146.stderr:50+0 records out 2024-03-09T15:35:43.630 INFO:tasks.workunit.client.0.smithi146.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.14371 s, 365 MB/s 2024-03-09T15:35:43.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-03-09T15:35:43.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:43.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.44644 2024-03-09T15:35:45.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:45.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.44644 2024-03-09T15:35:46.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:46.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.44644 2024-03-09T15:35:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.44644 2024-03-09T15:35:49.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:49.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.44644 2024-03-09T15:35:51.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.44644 2024-03-09T15:35:52.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:52.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.44644 2024-03-09T15:35:53.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:53.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.44644 2024-03-09T15:35:55.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:55.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.44644 2024-03-09T15:35:57.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:57.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.44644 2024-03-09T15:35:58.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:58.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.44644 2024-03-09T15:35:59.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:35:59.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.44644 2024-03-09T15:36:01.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:01.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.44644 2024-03-09T15:36:02.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:02.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.44644 2024-03-09T15:36:04.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:04.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.44644 2024-03-09T15:36:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.44644 2024-03-09T15:36:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.44644 2024-03-09T15:36:08.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:08.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.44644 2024-03-09T15:36:10.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:10.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.44644 2024-03-09T15:36:11.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:11.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.44644 2024-03-09T15:36:13.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:13.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.44644 2024-03-09T15:36:15.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:15.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.44644 2024-03-09T15:36:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.44644 2024-03-09T15:36:18.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:18.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.44644 2024-03-09T15:36:19.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:19.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.44644 2024-03-09T15:36:21.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:21.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.44644 2024-03-09T15:36:22.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.44644 2024-03-09T15:36:24.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:24.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.44644 2024-03-09T15:36:25.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:25.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.44644 2024-03-09T15:36:27.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:27.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.44644 2024-03-09T15:36:28.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:28.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.44644 2024-03-09T15:36:30.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:30.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.44644 2024-03-09T15:36:31.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:31.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.44644 2024-03-09T15:36:33.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:33.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.44644 2024-03-09T15:36:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.44644 2024-03-09T15:36:36.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:36.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.44644 2024-03-09T15:36:37.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:37.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.44644 2024-03-09T15:36:39.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:39.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.44644 2024-03-09T15:36:40.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:40.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.44644 2024-03-09T15:36:42.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T15:36:42.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.44644 2024-03-09T15:36:43.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.44644 2024-03-09T15:36:43.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-03-09T15:36:44.017 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 3 2024-03-09T15:36:44.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-09T15:35:39.176367+0000 49'26 49:56 [5] 5 [5] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.413543+0000 49'13 49:43 [3] 3 [3] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.1d 2 0 0 0 0 73400320 0 0 18 0 18 active+clean 2024-03-09T15:35:38.564665+0000 49'18 49:36 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.563957+0000 49'13 49:34 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.413144+0000 49'13 49:34 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:38.564190+0000 0'0 49:21 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.291 INFO:tasks.workunit.client.0.smithi146.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-09T15:35:39.058475+0000 49'26 49:47 [5] 5 [5] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.362450+0000 49'13 49:36 [6] 6 [6] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:38.362430+0000 0'0 49:23 [6] 6 [6] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-09T15:35:38.530417+0000 49'39 49:63 [4] 4 [4] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-09T15:35:38.530402+0000 49'52 49:76 [3] 3 [3] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-09T15:35:38.362466+0000 49'52 49:76 [4] 4 [4] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.446600+0000 49'13 49:40 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:38.413206+0000 0'0 49:22 [3] 3 [3] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:39.176142+0000 49'13 49:43 [5] 5 [5] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.589478+0000 49'13 49:34 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.958642+0000 49'13 49:42 [7] 7 [7] 7 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.958659+0000 49'13 49:33 [7] 7 [7] 7 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.564530+0000 49'13 49:33 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.530450+0000 49'13 49:33 [6] 6 [6] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:39.176257+0000 49'13 49:33 [5] 5 [5] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.589286+0000 49'13 49:36 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-09T15:35:37.446602+0000 49'26 49:48 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.530401+0000 49'13 49:33 [4] 4 [4] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:39.035405+0000 0'0 49:19 [2] 2 [2] 2 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-09T15:35:38.564372+0000 49'39 49:58 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:38.530404+0000 0'0 49:19 [6] 6 [6] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.589301+0000 49'13 49:35 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:39.175819+0000 0'0 49:19 [5] 5 [5] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:38.413203+0000 49'13 49:33 [3] 3 [3] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:35:38.564488+0000 0'0 49:19 [1] 1 [1] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T15:35:37.589432+0000 49'13 49:35 [0] 0 [0] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:44.293 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:36:44.294 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:36:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:44.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-03-09T15:36:44.428 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:36:44.428 INFO:tasks.workunit.client.0.smithi146.stdout: "osd_recovery_max_active": "0" 2024-03-09T15:36:44.429 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:36:44.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-03-09T15:36:44.672 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:36:44.672 INFO:tasks.workunit.client.0.smithi146.stdout: "osd_recovery_max_active_hdd": "3" 2024-03-09T15:36:44.672 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:36:44.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-03-09T15:36:44.941 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:36:44.941 INFO:tasks.workunit.client.0.smithi146.stdout: "osd_recovery_max_active_ssd": "10" 2024-03-09T15:36:44.941 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:36:44.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-03-09T15:36:44.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T15:36:44.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T15:36:44.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T15:36:44.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T15:36:44.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T15:36:45.383 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:45.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-09T15:36:45.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T15:36:47.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-03-09T15:36:47.396 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-03-09T15:36:47.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-03-09T15:36:47.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T15:36:47.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T15:36:47.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T15:36:47.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T15:36:47.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T15:36:47.668 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:47.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-09T15:36:47.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T15:36:49.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-03-09T15:36:49.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-03-09T15:36:49.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-03-09T15:36:49.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T15:36:49.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T15:36:49.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T15:36:49.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T15:36:49.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T15:36:49.951 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:49.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-03-09T15:36:49.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T15:36:51.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-03-09T15:36:51.965 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-03-09T15:36:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-03-09T15:36:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T15:36:51.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T15:36:51.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T15:36:51.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T15:36:51.968 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T15:36:52.238 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:52.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-03-09T15:36:52.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-03-09T15:36:52.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T15:36:45.902590+0000 49'26 52:66 [5,0,2] 5 [5,0,2] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.212573+0000 49'13 52:54 [3,0,7] 3 [3,0,7] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T15:36:45.366313+0000 49'26 52:55 [1,3,0] 1 [1,3,0] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-09T15:36:45.633395+0000 49'13 52:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:45.632051+0000 49'13 52:44 [1,6,2] 1 [1,6,2] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:45.633438+0000 0'0 52:33 [1,4,5] 1 [1,4,5] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T15:36:46.213232+0000 49'26 52:56 [5,4,3] 5 [5,4,3] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-09T15:36:45.859896+0000 49'13 52:47 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:45.859225+0000 49'13 52:46 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.507 INFO:tasks.workunit.client.0.smithi146.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-09T15:36:46.212485+0000 49'39 52:73 [4,2,6] 4 [4,2,6] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-09T15:36:45.782935+0000 49'52 52:86 [3,0,6] 3 [3,0,6] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-09T15:36:46.312466+0000 49'52 52:86 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.489697+0000 49'13 52:50 [0,7,1] 0 [0,7,1] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:45.733144+0000 0'0 52:33 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:45.902456+0000 49'13 52:53 [5,0,4] 5 [5,0,4] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.395871+0000 49'13 52:44 [0,3,4] 0 [0,3,4] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.564051+0000 49'13 52:53 [7,0,2] 7 [7,0,2] 7 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.212778+0000 49'13 52:42 [7,4,2] 7 [7,4,2] 7 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:45.633636+0000 49'13 52:43 [1,4,2] 1 [1,4,2] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.212605+0000 49'13 52:42 [6,0,3] 6 [6,0,3] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.158571+0000 49'13 52:42 [5,7,6] 5 [5,7,6] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.312317+0000 49'13 52:45 [0,1,3] 0 [0,1,3] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.508 INFO:tasks.workunit.client.0.smithi146.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T15:36:46.085542+0000 49'26 52:58 [0,1,6] 0 [0,1,6] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:45.954943+0000 49'13 52:43 [4,6,3] 4 [4,6,3] 4 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:46.395920+0000 0'0 52:30 [2,0,4] 2 [2,0,4] 2 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-09T15:36:45.633209+0000 49'39 52:68 [1,5,3] 1 [1,5,3] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:45.733214+0000 0'0 52:30 [6,0,7] 6 [6,0,7] 6 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.564001+0000 49'13 52:44 [0,4,5] 0 [0,4,5] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:45.732673+0000 0'0 52:30 [5,2,3] 5 [5,2,3] 5 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.395951+0000 49'13 52:42 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T15:36:45.195000+0000 0'0 52:31 [1,7,4] 1 [1,7,4] 1 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 periodic scrub scheduled @ 2024-03-10T15:35:36.686578+0000 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T15:36:46.085230+0000 49'13 52:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T15:35:36.686578+0000 0'0 2024-03-09T15:35:36.686578+0000 0 0 no scrub is scheduled 0 0 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:36:52.509 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:36:52.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-03-09T15:36:52.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-03-09T15:36:52.518 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-03-09T15:36:52.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-03-09T15:36:52.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-03-09T15:36:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-03-09T15:36:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66060"' 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66060' 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66063: /' 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66065"' 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66065' 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66070: /' 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66072"' 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66072' 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66077: /' 2024-03-09T15:36:52.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-03-09T15:36:52.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66079"' 2024-03-09T15:36:52.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66079' 2024-03-09T15:36:52.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66086"' 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66086' 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66091: /' 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66092"' 2024-03-09T15:36:52.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66092' 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66096: /' 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66102"' 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66102' 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-03-09T15:36:52.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66104"' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66104' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66106: /' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66083: /' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66108"' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66108' 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-03-09T15:36:52.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66111: /' 2024-03-09T15:36:52.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-03-09T15:36:52.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-03-09T15:36:52.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-03-09T15:36:52.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66120"' 2024-03-09T15:36:52.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66120' 2024-03-09T15:36:52.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66122"' 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66122' 2024-03-09T15:36:52.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66119: /' 2024-03-09T15:36:52.565 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-03-09T15:36:52.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66126: /' 2024-03-09T15:36:52.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66129"' 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66129' 2024-03-09T15:36:52.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-03-09T15:36:52.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-03-09T15:36:52.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-03-09T15:36:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66135"' 2024-03-09T15:36:52.581 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66135' 2024-03-09T15:36:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66133: /' 2024-03-09T15:36:52.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66138: /' 2024-03-09T15:36:52.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-03-09T15:36:52.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-03-09T15:36:52.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-03-09T15:36:52.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66146"' 2024-03-09T15:36:52.596 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66146' 2024-03-09T15:36:52.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-03-09T15:36:52.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-03-09T15:36:52.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66143: /' 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66151"' 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66151' 2024-03-09T15:36:52.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66157"' 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66157' 2024-03-09T15:36:52.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-03-09T15:36:52.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66154: /' 2024-03-09T15:36:52.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-03-09T15:36:52.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-03-09T15:36:52.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66162"' 2024-03-09T15:36:52.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66162' 2024-03-09T15:36:52.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-03-09T15:36:52.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-03-09T15:36:52.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66168"' 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66168' 2024-03-09T15:36:52.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66165: /' 2024-03-09T15:36:52.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66161: /' 2024-03-09T15:36:52.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-03-09T15:36:52.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-03-09T15:36:52.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66174"' 2024-03-09T15:36:52.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66174' 2024-03-09T15:36:52.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-03-09T15:36:52.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66173: /' 2024-03-09T15:36:52.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-03-09T15:36:52.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-03-09T15:36:52.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66181: /' 2024-03-09T15:36:52.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-03-09T15:36:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-03-09T15:36:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66191"' 2024-03-09T15:36:52.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66191' 2024-03-09T15:36:52.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66187: /' 2024-03-09T15:36:52.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-03-09T15:36:52.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-03-09T15:36:52.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66197"' 2024-03-09T15:36:52.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66197' 2024-03-09T15:36:52.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66200: /' 2024-03-09T15:36:52.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-03-09T15:36:52.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66205"' 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66205' 2024-03-09T15:36:52.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-03-09T15:36:52.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66208: /' 2024-03-09T15:36:52.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66215"' 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66215' 2024-03-09T15:36:52.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-03-09T15:36:52.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66214: /' 2024-03-09T15:36:52.724 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-03-09T15:36:52.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-03-09T15:36:52.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66223"' 2024-03-09T15:36:52.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66223' 2024-03-09T15:36:52.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66229: /' 2024-03-09T15:36:52.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-03-09T15:36:52.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-03-09T15:36:52.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-03-09T15:36:52.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66241"' 2024-03-09T15:36:52.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66241' 2024-03-09T15:36:52.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-03-09T15:36:52.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66244: /' 2024-03-09T15:36:52.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-03-09T15:36:52.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-03-09T15:36:52.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66263"' 2024-03-09T15:36:52.763 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66263' 2024-03-09T15:36:52.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66266: /' 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66275"' 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66275' 2024-03-09T15:36:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-03-09T15:36:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-03-09T15:36:52.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-03-09T15:36:52.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66284: /' 2024-03-09T15:36:52.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66291: /' 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66304"' 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66304' 2024-03-09T15:36:52.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-03-09T15:36:52.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66325"' 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66325' 2024-03-09T15:36:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66322: /' 2024-03-09T15:36:52.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66344"' 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66344' 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66334: /' 2024-03-09T15:36:52.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-03-09T15:36:52.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-03-09T15:36:52.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-03-09T15:36:52.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66351"' 2024-03-09T15:36:52.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66351' 2024-03-09T15:36:52.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T15:36:52.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66367: /' 2024-03-09T15:36:52.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-03-09T15:36:52.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66349: /' 2024-03-09T15:36:52.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66379"' 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66379' 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 66060 66065 66072 66079 66086 66092 66102 66104 66108 66120 66122 66129 66135 66146 66151 66157 66162 66168 66174 66191 66197 66205 66215 66223 66241 66263 66275 66304 66325 66344 66351 66379' 2024-03-09T15:36:52.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-03-09T15:36:52.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:36:52.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66060 2024-03-09T15:36:52.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-03-09T15:36:52.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-03-09T15:36:52.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66398: /' 2024-03-09T15:36:59.843 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-09T15:36:59.844 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:36:59.845 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: instructing pg 1.a on osd.1 to scrub 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: dumped pgs 2024-03-09T15:36:59.846 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-03-09T15:35:36.686578+0000 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:36:59.847 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-09T15:36:59.848 INFO:tasks.workunit.client.0.smithi146.stderr:66133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:36:59.849 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:36:59.849 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:36:59.849 INFO:tasks.workunit.client.0.smithi146.stderr:66133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:36:59.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.374 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: instructing pg 1.b on osd.1 to scrub 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: dumped pgs 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-03-09T15:37:00.375 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.376 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66138: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:37:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: instructing pg 1.0 on osd.7 to scrub 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: dumped pgs 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-03-09T15:37:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:66063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:00.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:00.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:00.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:00.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:00.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:00.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66065 2024-03-09T15:37:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-03-09T15:37:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-03-09T15:37:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-09T15:37:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: instructing pg 1.e on osd.1 to scrub 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: dumped pgs 2024-03-09T15:37:00.664 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:00.665 INFO:tasks.workunit.client.0.smithi146.stderr:66161: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:00.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:00.695 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: instructing pg 1.8 on osd.5 to scrub 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: dumped pgs 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:00.696 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:66111: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:00.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:00.715 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-03-09T15:37:00.715 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-03-09T15:37:00.715 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: instructing pg 1.f on osd.6 to scrub 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-03-09T15:37:00.716 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: dumped pgs 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:00.717 INFO:tasks.workunit.client.0.smithi146.stderr:66165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-09T15:37:00.718 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:00.718 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:00.718 INFO:tasks.workunit.client.0.smithi146.stderr:66165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:00.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: instructing pg 1.1a on osd.1 to scrub 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: dumped pgs 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:66291: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: instructing pg 1.10 on osd.5 to scrub 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: dumped pgs 2024-03-09T15:37:01.133 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.134 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.135 INFO:tasks.workunit.client.0.smithi146.stderr:66173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.197 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-03-09T15:37:01.197 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-03-09T15:37:01.197 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:37:01.197 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: instructing pg 1.3 on osd.4 to scrub 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-03-09T15:37:01.198 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: dumped pgs 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.199 INFO:tasks.workunit.client.0.smithi146.stderr:66083: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:37:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:66083: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.221 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: instructing pg 1.7 on osd.6 to scrub 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: dumped pgs 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.222 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.223 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.244 INFO:tasks.workunit.client.0.smithi146.stderr:66119: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-03-09T15:37:01.244 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-03-09T15:37:01.244 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-09T15:37:01.244 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: instructing pg 1.11 on osd.0 to scrub 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: dumped pgs 2024-03-09T15:37:01.245 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:66181: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-03-09T15:37:01.264 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: instructing pg 1.1 on osd.5 to scrub 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: dumped pgs 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.265 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:66070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66072 2024-03-09T15:37:01.351 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-03-09T15:37:01.351 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.352 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: instructing pg 1.19 on osd.3 to scrub 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: dumped pgs 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-03-09T15:37:01.353 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:66284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-09T15:37:01.437 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: instructing pg 1.c on osd.3 to scrub 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: dumped pgs 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.438 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.439 INFO:tasks.workunit.client.0.smithi146.stderr:66143: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-03-09T15:37:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: instructing pg 1.13 on osd.4 to scrub 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: dumped pgs 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.448 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:66200: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.489 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-03-09T15:37:01.489 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-03-09T15:37:01.489 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-09T15:37:01.489 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.489 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: instructing pg 1.15 on osd.1 to scrub 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.490 INFO:tasks.workunit.client.0.smithi146.stderr:66214: dumped pgs 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:66214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-03-09T15:37:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-03-09T15:37:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-09T15:37:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: instructing pg 1.12 on osd.0 to scrub 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: dumped pgs 2024-03-09T15:37:01.542 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:66187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.648 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: instructing pg 1.6 on osd.6 to scrub 2024-03-09T15:37:01.649 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: dumped pgs 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-09T15:37:01.650 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:66106: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:66106: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.817 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-03-09T15:37:01.817 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-03-09T15:37:01.818 INFO:tasks.workunit.client.0.smithi146.stderr:66398: instructing pg 1.1f on osd.5 to scrub 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: dumped pgs 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-09T15:37:01.819 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:66398: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:66398: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.823 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: instructing pg 1.1c on osd.0 to scrub 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: dumped pgs 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-03-09T15:37:01.824 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.825 INFO:tasks.workunit.client.0.smithi146.stderr:66334: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.850 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: instructing pg 1.1b on osd.0 to scrub 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: dumped pgs 2024-03-09T15:37:01.851 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.852 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.853 INFO:tasks.workunit.client.0.smithi146.stderr:66322: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.866 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-03-09T15:37:01.866 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: instructing pg 1.1d on osd.1 to scrub 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.867 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: dumped pgs 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-09T15:37:01.868 INFO:tasks.workunit.client.0.smithi146.stderr:66349: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.869 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.869 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.869 INFO:tasks.workunit.client.0.smithi146.stderr:66349: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.880 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: instructing pg 1.5 on osd.4 to scrub 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: dumped pgs 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.881 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.882 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.883 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.883 INFO:tasks.workunit.client.0.smithi146.stderr:66096: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.899 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-03-09T15:37:01.899 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-03-09T15:37:01.899 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-09T15:37:01.899 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: instructing pg 1.14 on osd.2 to scrub 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-03-09T15:37:01.900 INFO:tasks.workunit.client.0.smithi146.stderr:66208: dumped pgs 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:36:58.097433+0000 '>' '' 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.901 INFO:tasks.workunit.client.0.smithi146.stderr:66208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.948 INFO:tasks.workunit.client.0.smithi146.stderr:lect(.pgid=="1.b") | .last_scrub_stamp' 2024-03-09T15:37:01.948 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.948 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.948 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-09T15:37:01.948 INFO:tasks.workunit.client.0.smithi146.stderr:66138: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-09T15:37:01.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-09T15:37:01.986 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-03-09T15:37:01.986 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: instructing pg 1.1e on osd.3 to scrub 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.987 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: dumped pgs 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.988 INFO:tasks.workunit.client.0.smithi146.stderr:66367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-03-09T15:37:01.989 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: instructing pg 1.16 on osd.6 to scrub 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: dumped pgs 2024-03-09T15:37:01.990 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-09T15:37:01.991 INFO:tasks.workunit.client.0.smithi146.stderr:66229: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:36:58.426282+0000 '>' '' 2024-03-09T15:37:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:66229: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.993 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-03-09T15:37:01.994 INFO:tasks.workunit.client.0.smithi146.stderr:66077: instructing pg 1.2 on osd.0 to scrub 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: dumped pgs 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-03-09T15:37:01.995 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:66077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66079 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:01.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66086 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.997 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: instructing pg 1.9 on osd.1 to scrub 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: dumped pgs 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.998 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:01.999 INFO:tasks.workunit.client.0.smithi146.stderr:66126: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-09T15:37:02.000 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: instructing pg 1.d on osd.7 to scrub 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-03-09T15:37:02.001 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: dumped pgs 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.002 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.003 INFO:tasks.workunit.client.0.smithi146.stderr:66154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-09T15:37:02.003 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:02.003 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:02.003 INFO:tasks.workunit.client.0.smithi146.stderr:66154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:02.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-03-09T15:37:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-03-09T15:37:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: instructing pg 1.4 on osd.3 to scrub 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-03-09T15:37:02.008 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: dumped pgs 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-09T15:37:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:66091: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:66091: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66092 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66102 2024-03-09T15:37:02.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66104 2024-03-09T15:37:02.042 INFO:tasks.workunit.client.0.smithi146.stderr:lect(.pgid=="1.7") | .last_scrub_stamp' 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:66119: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-09T15:37:02.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-09T15:37:02.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-03-09T15:37:02.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-03-09T15:37:02.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66108 2024-03-09T15:37:02.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66120 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66122 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66129 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-09T15:37:02.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66135 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66146 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66151 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66157 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66162 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66168 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66174 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66191 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66197 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66205 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66215 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66223 2024-03-09T15:37:02.098 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-03-09T15:37:02.098 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-03-09T15:37:02.098 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-09T15:37:02.098 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: instructing pg 1.17 on osd.0 to scrub 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:02.099 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: dumped pgs 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-09T15:37:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:66244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:66244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66241 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T15:37:02.146 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: instructing pg 1.18 on osd.5 to scrub 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: dumped pgs 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T15:35:36.686578+0000 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T15:37:02.147 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T15:35:36.686578+0000 '>' '' 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:66266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:37:02.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66263 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66275 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66304 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66325 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66344 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66351 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66379 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-03-09T15:37:02.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.5.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.6.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.1.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.0.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.4.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.3.pid 2024-03-09T15:37:02.152 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.7.pid 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:td/osd-recovery-scrub/osd.2.pid' 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53204 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:61116 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:62492 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:55610 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:53288 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:54235 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:59740 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:58362 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:63868 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:56986' 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53204 61116 62492 55610 53288 54235 59740 58362 63868 56986 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-03-09T15:37:02.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:37:02.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:37:02.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:37:02.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:37:02.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:37:07.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:37:07.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-03-09T15:37:07.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-03-09T15:37:07.461 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T15:37:07.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-03-09T15:37:07.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-03-09T15:37:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-03-09T15:37:07.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-03-09T15:37:07.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-03-09T15:37:07.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-03-09T15:37:07.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-03-09T15:37:07.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-03-09T15:37:07.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-03-09T15:37:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-03-09T15:37:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-03-09T15:37:07.495 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T15:37:07.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-03-09T15:37:07.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-03-09T15:37:07.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-03-09T15:37:07.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-03-09T15:37:07.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-03-09T15:37:07.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-03-09T15:37:07.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-03-09T15:37:07.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-03-09T15:37:07.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T15:37:07.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-03-09T15:37:07.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-03-09T15:37:07.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:37:07.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:37:07.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:37:07.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:37:07.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:37:07.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:37:07.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:37:07.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:37:07.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:37:07.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:37:07.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:37:07.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:37:07.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:37:07.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T15:37:07.562 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:37:07.562 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:07.562 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:37:07.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44644 2024-03-09T15:37:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:37:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:37:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-03-09T15:37:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-03-09T15:37:07.563 INFO:tasks.workunit.client.0.smithi146.stdout:TEST PASSED 2024-03-09T15:37:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-03-09T15:37:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:37:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:37:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:37:07.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:37:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:37:07.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:37:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:37:07.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:37:07.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:37:07.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:37:07.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:37:07.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:37:07.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T15:37:07.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T15:37:07.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:37:07.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:07.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44644 2024-03-09T15:37:07.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44644 2024-03-09T15:37:07.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:37:07.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:37:07.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T15:37:07.574 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T15:37:07.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T15:37:07.602 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-03-09T15:37:07.602 DEBUG:teuthology.orchestra.run.smithi146:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-09T15:37:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T15:37:07.668 INFO:tasks.workunit.client.0.smithi146.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T15:37:07.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-03-09T15:37:07.669 INFO:tasks.workunit.client.0.smithi146.stdout:This test is disabled 2024-03-09T15:37:07.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-03-09T15:37:07.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T15:37:07.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:37:07.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:37:07.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:37:07.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:37:07.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T15:37:07.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-03-09T15:37:07.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:37:07.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:07.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69541 2024-03-09T15:37:07.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69541 2024-03-09T15:37:07.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:37:07.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:37:07.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T15:37:07.678 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T15:37:07.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T15:37:07.744 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-03-09T15:37:07.745 DEBUG:teuthology.orchestra.run.smithi146:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-09T15:37:07.804 INFO:tasks.workunit.client.0.smithi146.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-09T15:37:07.804 INFO:tasks.workunit.client.0.smithi146.stderr:++ TIMEOUT=300 2024-03-09T15:37:07.804 INFO:tasks.workunit.client.0.smithi146.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ MAX_TIMEOUT=15 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ PG_NUM=4 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ TMPDIR=/tmp 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ type xmlstarlet 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ XMLSTARLET=xmlstarlet 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:+++ uname 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' Linux = FreeBSD ']' 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ SED=sed 2024-03-09T15:37:07.805 INFO:tasks.workunit.client.0.smithi146.stderr:++ AWK=awk 2024-03-09T15:37:07.806 INFO:tasks.workunit.client.0.smithi146.stderr:+++ stty -a 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:+++ head -1 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:++ termwidth= 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:++ DIFFCOLOPTS='-y ' 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:++ KERNCORE=kernel.core_pattern 2024-03-09T15:37:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:++ EXTRA_OPTS= 2024-03-09T15:37:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:++ test '' = TESTS 2024-03-09T15:37:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-09T15:37:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:++ uname 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' Linux = FreeBSD ']' 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:+ use_ec_overwrite=true 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:+ getjson=no 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:+ jqfilter='def walk(f): 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T15:37:07.811 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T15:37:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ main osd-scrub-repair 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ local dir=td/osd-scrub-repair 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ shopt -s -o xtrace 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:37:07.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-03-09T15:37:07.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none ' 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-03-09T15:37:07.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_bluestore_basic 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_bluestore_failed 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_bluestore_scrub 2024-03-09T15:37:07.821 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_bluestore_tag 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_erasure_coded_appends 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_and_repair_lrc_appends 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_and_repair_replicated 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_scrub_erasure_appends 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_scrub_replicated 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_corrupt_snapset_scrub_rep 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_list_missing_erasure_coded_appends 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_list_missing_erasure_coded_overwrites 2024-03-09T15:37:07.822 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_periodic_scrub_replicated 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_repair_stats 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_repair_stats_ec 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_request_scrub_priority 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_scrub_warning 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_skip_non_repair_during_recovery 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_unfound_erasure_coded_appends 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:TEST_unfound_erasure_coded_overwrites' 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:37:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:37:07.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:37:07.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:37:07.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:37:07.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:37:07.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:37:07.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:37:07.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:37:07.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:37:07.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:37:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:37:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:37:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:37:07.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:37:07.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:07.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:07.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:37:07.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:07.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:37:07.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:37:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T15:37:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T15:37:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:37:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:37:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:08.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:08.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T15:37:08.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:37:08.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:37:08.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:37:08.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:37:08.403 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:37:08.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:37:08.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:37:08.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:37:09.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:37:09.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:37:09.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:37:09.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:37:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da666f12-424e-4365-a34b-4d62f1f957ce 2024-03-09T15:37:09.331 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 da666f12-424e-4365-a34b-4d62f1f957ce 2024-03-09T15:37:09.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 da666f12-424e-4365-a34b-4d62f1f957ce' 2024-03-09T15:37:09.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:37:09.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlguxl+d2lFBAATlyNktBq3TIyf88pw2q+nA== 2024-03-09T15:37:09.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlguxl+d2lFBAATlyNktBq3TIyf88pw2q+nA=="}' 2024-03-09T15:37:09.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da666f12-424e-4365-a34b-4d62f1f957ce -i td/osd-scrub-repair/0/new.json 2024-03-09T15:37:09.552 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:37:09.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:37:09.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAlguxl+d2lFBAATlyNktBq3TIyf88pw2q+nA== --osd-uuid da666f12-424e-4365-a34b-4d62f1f957ce 2024-03-09T15:37:09.593 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.594+0000 7f184f02a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:09.603 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.603+0000 7f184f02a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:09.604 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.605+0000 7f184f02a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:09.605 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.605+0000 7f184f02a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:37:09.605 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.605+0000 7f184f02a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:37:09.605 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.605+0000 7f184f02a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:37:09.711 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:09.712+0000 7f184f02a640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:37:15.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:37:15.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:37:15.225 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:37:15.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:37:15.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:37:15.579 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:37:15.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:37:15.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T15:37:15.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:37:15.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:37:15.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:37:15.607 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:15.608+0000 7f2e17c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:15.609 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:15.610+0000 7f2e17c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:15.610 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:15.611+0000 7f2e17c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:37:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:37:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:37:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:16.130 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:16.131+0000 7f2e17c22640 -1 Falling back to public interface 2024-03-09T15:37:16.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:17.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:17.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:17.160 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:37:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:37:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:17.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:18.426 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:18.426+0000 7f2e17c22640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:37:18.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:18.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:18.441 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:37:18.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:37:18.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:18.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:18.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:19.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:19.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:19.852 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:37:19.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:37:19.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:19.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:20.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:21.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:21.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:21.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:37:21.141 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:37:21.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:21.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:21.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:22.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:22.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:22.425 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:37:22.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:37:22.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:22.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:22.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:23.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:23.706 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:37:23.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:23.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:37:23.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:23.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:24.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:24.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:24.992 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:37:24.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:37:24.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:24.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:25.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:26.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:26.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:26.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:37:26.341 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:37:26.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:26.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:26.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:37:27.622 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:37:27.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:27.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:27.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:28.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:28.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:28.995 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:37:28.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:37:28.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:28.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:29.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:30.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:30.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:30.383 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:37:30.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:37:30.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:30.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:30.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:31.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:31.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:31.863 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:37:31.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:37:31.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:31.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:32.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:33.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:33.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:33.509 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:37:33.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:37:33.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:33.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:34.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:35.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:35.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:35.372 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:37:35.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:37:35.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:35.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:36.699 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:37:36.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:37:36.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:36.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:37.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:38.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:38.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:38.400 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:37:38.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:37:38.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:38.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:39.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:40.111 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:37:40.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:37:40.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:40.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:37:40.390 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3120809584,v1:127.0.0.1:6803/3120809584] [v2:127.0.0.1:6804/3120809584,v1:127.0.0.1:6805/3120809584] exists,up da666f12-424e-4365-a34b-4d62f1f957ce 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:37:40.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:40.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:37:40.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17b461af-80db-44dc-8fd7-ad9e89785f65 2024-03-09T15:37:40.397 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 17b461af-80db-44dc-8fd7-ad9e89785f65 2024-03-09T15:37:40.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 17b461af-80db-44dc-8fd7-ad9e89785f65' 2024-03-09T15:37:40.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:37:40.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEguxlSvuAGBAAJBD/b7t53/lNvHEaDW7rWA== 2024-03-09T15:37:40.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEguxlSvuAGBAAJBD/b7t53/lNvHEaDW7rWA=="}' 2024-03-09T15:37:40.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17b461af-80db-44dc-8fd7-ad9e89785f65 -i td/osd-scrub-repair/1/new.json 2024-03-09T15:37:40.819 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:37:40.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:37:40.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBEguxlSvuAGBAAJBD/b7t53/lNvHEaDW7rWA== --osd-uuid 17b461af-80db-44dc-8fd7-ad9e89785f65 2024-03-09T15:37:40.860 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.860+0000 7f636c9ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:40.887 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.887+0000 7f636c9ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:40.888 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.888+0000 7f636c9ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:40.888 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.888+0000 7f636c9ac640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:37:40.889 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.888+0000 7f636c9ac640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:37:40.889 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.888+0000 7f636c9ac640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:37:40.953 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:40.953+0000 7f636c9ac640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:37:46.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:37:46.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:37:46.833 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:37:46.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:37:46.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:37:47.191 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:37:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:37:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T15:37:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:37:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:37:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:37:47.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:47.219+0000 7f9a25024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:47.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:47.222+0000 7f9a25024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:47.222 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:47.223+0000 7f9a25024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:37:47.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:37:47.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:37:47.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:37:47.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:37:47.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:37:47.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:47.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:37:47.469 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:37:47.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:47.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:47.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:47.744 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:47.745+0000 7f9a25024640 -1 Falling back to public interface 2024-03-09T15:37:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:48.739 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:37:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:37:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:49.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:50.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:50.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:50.023 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:37:50.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:37:50.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:50.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:50.065 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:37:50.066+0000 7f9a25024640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:37:50.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:51.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:51.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:51.367 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:37:51.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:37:51.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:51.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:51.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:52.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:52.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:52.748 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:37:52.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:37:52.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:52.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:53.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:54.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:54.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:54.031 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:37:54.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:37:54.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:54.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:54.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:55.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:55.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:55.316 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:37:55.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:37:55.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:55.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:55.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:56.721 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:37:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:37:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:57.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:58.053 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:37:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:37:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:58.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:37:59.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:37:59.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:37:59.337 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:37:59.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:37:59.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:37:59.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:37:59.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:00.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:00.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:00.720 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:38:00.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:38:00.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:00.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:01.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:02.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:02.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:02.006 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:38:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:38:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:02.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:02.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:03.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:03.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:03.409 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:38:03.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:38:03.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:03.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:03.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:04.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:04.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:04.778 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:38:04.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:38:04.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:04.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:05.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:06.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:06.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:06.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:38:06.485 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:38:06.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:06.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:06.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:07.773 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:08.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:09.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:09.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:09.488 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:38:09.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:38:09.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:09.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:10.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:11.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:11.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:11.078 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:38:11.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:38:11.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:11.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:11.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:12.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:12.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:12.556 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:38:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:38:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:12.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:13.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:13.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:13.899 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:38:13.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:38:13.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:13.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:14.180 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1830564795,v1:127.0.0.1:6811/1830564795] [v2:127.0.0.1:6812/1830564795,v1:127.0.0.1:6813/1830564795] exists,up 17b461af-80db-44dc-8fd7-ad9e89785f65 2024-03-09T15:38:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:38:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:38:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:38:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-03-09T15:38:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T15:38:14.459 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T15:38:14.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T15:38:14.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T15:38:14.873 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T15:38:14.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:38:15.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T15:38:17.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-03-09T15:38:17.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:38:17.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:38:17.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:38:17.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:38:17.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:38:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:38:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:38:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:38:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:38:17.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:38:18.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:38:18.028 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T15:38:18.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:38:18.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:18.028 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:38:18.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-09T15:38:18.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-09T15:38:18.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-09T15:38:18.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:18.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:38:18.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T15:38:18.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T15:38:18.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-03-09T15:38:18.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:18.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-09T15:38:18.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:18.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:38:18.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-09T15:38:18.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:18.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-09T15:38:18.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-09T15:38:18.239 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836493 2024-03-09T15:38:18.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:18.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-09T15:38:18.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:38:19.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:38:19.510 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:19.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-09T15:38:19.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:19.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T15:38:19.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:19.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:38:19.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T15:38:19.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:19.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T15:38:19.793 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672966 2024-03-09T15:38:19.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T15:38:19.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:38:20.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-09T15:38:20.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:38:20.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:20.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:38:20.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:38:20.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:38:20.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:38:20.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:20.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:21.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:38:21.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:38:21.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:38:21.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-03-09T15:38:21.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T15:38:21.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-09T15:38:21.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T15:38:21.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T15:38:21.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T15:38:21.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T15:38:21.533 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T15:38:21.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T15:38:21.951 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T15:38:21.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T15:38:21.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T15:38:21.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:38:22.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T15:38:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:38:22.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:38:22.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T15:38:22.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:38:22.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:38:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:38:22.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:38:22.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:38:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:38:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:38:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:38:23.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:38:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T15:38:24.606 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T15:38:25.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T15:38:25.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:38:25.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:38:25.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:38:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:38:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:38:25.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:38:25.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:38:25.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:38:25.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:38:25.154 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:25.154+0000 7fb7894bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:25.156 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:25.157+0000 7fb7894bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:25.157 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:25.158+0000 7fb7894bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:25.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:38:25.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:38:25.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:38:25.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:38:25.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:38:25.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:25.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:38:25.406 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:38:25.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:25.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:38:25.680 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:25.681+0000 7fb7894bd640 -1 Falling back to public interface 2024-03-09T15:38:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:26.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:26.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:26.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:38:26.686 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:38:26.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:26.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:38:26.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:27.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:27.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:27.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:38:27.961 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:38:27.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:27.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:38:28.049 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:28.049+0000 7fb7894bd640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:38:28.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:29.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4174704585,v1:127.0.0.1:6803/4174704585] [v2:127.0.0.1:6804/4174704585,v1:127.0.0.1:6805/4174704585] exists,up da666f12-424e-4365-a34b-4d62f1f957ce 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:38:29.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:38:29.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:38:29.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:38:29.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:38:29.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:38:29.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:38:29.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:38:30.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:38:30.100 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T15:38:30.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:38:30.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:30.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:38:30.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-09T15:38:30.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-09T15:38:30.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-09T15:38:30.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:30.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:38:30.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-09T15:38:30.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-09T15:38:30.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-03-09T15:38:30.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:30.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-09T15:38:30.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:30.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:38:30.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-09T15:38:30.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:30.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-09T15:38:30.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-09T15:38:30.307 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149698 2024-03-09T15:38:30.307 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:30.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-09T15:38:30.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:38:31.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:38:31.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:31.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-09T15:38:31.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:31.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-09T15:38:31.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:31.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:38:31.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-09T15:38:31.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:31.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-09T15:38:31.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-09T15:38:31.866 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672970 2024-03-09T15:38:31.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:38:32.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-09T15:38:32.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:38:32.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:32.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:38:32.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:38:32.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:38:32.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:38:32.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:32.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:33.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T15:38:33.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:33.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:33.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:33.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T15:38:33.875 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:33.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:34.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:14.827653+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:34.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:38:35.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:38:35.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:35.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:35.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:35.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:35.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:35.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:35.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:14.827653+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:35.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:38:36.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:38:36.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:36.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:36.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:36.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:36.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:36.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:36.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:14.827653+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:36.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:38:37.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:38:37.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:37.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:37.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:37.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:37.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:37.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:37.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:14.827653+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:37.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:38.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:39.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:14.827653+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:39.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:38:40.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:38:40.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:38:40.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T15:38:40.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T15:38:40.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:38:40.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:38:40.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T15:38:40.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:38:34.132123+0000 '>' 2024-03-09T15:38:14.827653+0000 2024-03-09T15:38:40.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:38:40.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:38:40.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:38:40.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:38:40.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:38:40.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T15:38:42.044 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T15:38:42.044 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:38:42.045 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:38:42.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:38:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:38:42.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:38:42.314 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:38:42.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:38:42.342 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:42.341+0000 7f68cdc1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:42.344 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:42.344+0000 7f68cdc1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:42.345 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:42.345+0000 7f68cdc1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:38:42.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:38:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:38:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:42.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:42.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:42.876 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:42.876+0000 7f68cdc1e640 -1 Falling back to public interface 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:43.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:44.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:38:45.062 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:45.062+0000 7f68cdc1e640 -1 osd.1 28 log_to_monitors true 2024-03-09T15:38:45.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:38:45.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:38:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:38:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:38:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:38:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 32 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3619796048,v1:127.0.0.1:6811/3619796048] [v2:127.0.0.1:6812/3619796048,v1:127.0.0.1:6813/3619796048] exists,up 17b461af-80db-44dc-8fd7-ad9e89785f65 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:38:45.474 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:38:45.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:38:45.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:38:45.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:38:45.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:38:45.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:38:45.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:38:45.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:38:45.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:38:45.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:38:45.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:38:45.831 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T15:38:45.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:38:45.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:45.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:38:46.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-09T15:38:46.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-09T15:38:46.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-03-09T15:38:46.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:38:46.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-03-09T15:38:46.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:46.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:38:46.227 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-03-09T15:38:46.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:46.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-09T15:38:46.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-03-09T15:38:46.229 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149702 2024-03-09T15:38:46.229 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:46.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-03-09T15:38:46.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:38:47.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:38:47.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:38:47.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-09T15:38:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:38:47.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-09T15:38:47.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:38:47.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:38:47.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-09T15:38:47.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:38:47.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-09T15:38:47.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-09T15:38:47.823 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 137438953474 2024-03-09T15:38:47.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:38:48.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-09T15:38:48.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:38:48.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:48.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:38:48.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:38:48.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T15:38:48.744 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:38:48.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:48.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:38:49.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:38:49.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:38:49.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:38:49.180 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:38:49.180 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:38:49.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:38:49.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:38:49.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:38:49.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:38:49.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:38:49.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:38:49.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:38:49.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:38:49.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:38:49.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:38:49.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:38:49.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:38:49.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:38:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:38:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:38:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:38:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:38:49.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:38:49.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:38:49.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:38:49.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:38:49.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:38:49.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:38:49.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:38:49.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:38:49.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:49.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:38:49.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:38:49.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:38:49.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:38:49.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:38:49.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:38:49.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:38:49.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:38:49.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:38:49.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:38:49.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:38:49.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:38:49.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:38:49.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:38:49.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:38:49.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:38:49.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:38:49.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:38:49.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:38:49.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:38:49.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:38:49.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:38:49.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:38:49.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:38:49.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:38:49.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:38:49.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:38:49.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:38:49.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:38:49.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:38:49.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:38:49.953 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:38:49.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:38:49.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:38:49.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:38:49.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-03-09T15:38:49.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-03-09T15:38:49.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-09T15:38:49.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-09T15:38:49.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-09T15:38:51.497 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 508b1fba-ac18-4873-b896-220c97084676 2024-03-09T15:38:51.720 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:38:51.764 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.764+0000 7f1c8901c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:51.770 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.770+0000 7f1c8901c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:51.771 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.771+0000 7f1c8901c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:51.772 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.771+0000 7f1c8901c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:38:51.772 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.771+0000 7f1c8901c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:38:51.772 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.771+0000 7f1c8901c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:38:51.854 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:51.853+0000 7f1c8901c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:38:57.368 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:38:57.739 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:38:57.771 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:57.771+0000 7fbcf0e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:57.785 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:57.785+0000 7fbcf0e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:57.786+0000 7fbcf0e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:38:58.051 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:38:58.313 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:38:58.314+0000 7fbcf0e15640 -1 Falling back to public interface 2024-03-09T15:38:59.324 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:39:00.600 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:39:00.643 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:00.643+0000 7fbcf0e15640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:39:01.902 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:39:03.223 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:39:04.587 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:39:05.869 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:39:07.194 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:39:08.595 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:39:09.872 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:39:11.225 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:39:12.567 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:39:14.152 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:39:15.875 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:39:17.590 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:39:19.465 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:39:20.943 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:39:22.679 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:39:22.971 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1272965402,v1:127.0.0.1:6803/1272965402] [v2:127.0.0.1:6804/1272965402,v1:127.0.0.1:6805/1272965402] exists,up 508b1fba-ac18-4873-b896-220c97084676 2024-03-09T15:39:22.975 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 20288f54-02ec-4d06-9df1-df9433a9bcbe 2024-03-09T15:39:23.312 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:39:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.358+0000 7f427180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:23.397 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.397+0000 7f427180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:23.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.398+0000 7f427180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:23.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.398+0000 7f427180d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:39:23.400 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.398+0000 7f427180d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:39:23.400 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.398+0000 7f427180d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:39:23.473 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:23.472+0000 7f427180d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:39:28.967 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:39:29.315 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:39:29.346 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:29.346+0000 7f0c7bd2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:29.348 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:29.348+0000 7f0c7bd2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:29.349 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:29.350+0000 7f0c7bd2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:29.594 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:39:29.875 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:29.876+0000 7f0c7bd2a640 -1 Falling back to public interface 2024-03-09T15:39:30.866 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:39:32.149 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:39:32.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:32.193+0000 7f0c7bd2a640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:39:33.460 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:39:34.784 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:39:36.161 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:39:37.446 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:39:38.784 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:39:40.121 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:39:41.402 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:39:42.719 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:39:44.004 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:39:45.479 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:39:47.106 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:39:48.637 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:39:50.363 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:39:51.970 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:39:53.287 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:39:54.740 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:39:55.010 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3310268205,v1:127.0.0.1:6811/3310268205] [v2:127.0.0.1:6812/3310268205,v1:127.0.0.1:6813/3310268205] exists,up 20288f54-02ec-4d06-9df1-df9433a9bcbe 2024-03-09T15:39:55.014 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 d828b13a-54cf-4a4d-9963-6cba54fa6c22 2024-03-09T15:39:55.449 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:39:55.495 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.494+0000 7f073f80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:55.534 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.533+0000 7f073f80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:55.535 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.534+0000 7f073f80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:39:55.535 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.535+0000 7f073f80d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:39:55.535 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.535+0000 7f073f80d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:39:55.536 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.535+0000 7f073f80d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:39:55.642 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:39:55.641+0000 7f073f80d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:40:01.339 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:40:01.700 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:40:01.731 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:01.731+0000 7ff500819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:01.732 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:01.733+0000 7ff500819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:01.733 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:01.734+0000 7ff500819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:01.974 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:40:02.270 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:02.271+0000 7ff500819640 -1 Falling back to public interface 2024-03-09T15:40:03.247 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:40:04.535 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:40:04.635 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:04.635+0000 7ff500819640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:40:05.822 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:40:07.270 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:40:08.613 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:40:09.894 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:40:11.178 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:40:12.478 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:40:13.760 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:40:15.135 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:40:16.460 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:40:17.993 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:40:19.575 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:40:21.522 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:40:22.988 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:40:24.510 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:40:26.129 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:40:26.421 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3824255338,v1:127.0.0.1:6819/3824255338] [v2:127.0.0.1:6820/3824255338,v1:127.0.0.1:6821/3824255338] exists,up d828b13a-54cf-4a4d-9963-6cba54fa6c22 2024-03-09T15:40:26.873 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T15:40:28.597 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T15:40:30.150 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672969 2024-03-09T15:40:30.423 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509445 2024-03-09T15:40:32.912 INFO:tasks.workunit.client.0.smithi146.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-03-09T15:40:32.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-03-09T15:40:32.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-03-09T15:40:32.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-03-09T15:40:33.286 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T15:40:33.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-09T15:40:33.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:40:33.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:40:33.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:40:33.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:40:33.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:40:33.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:40:33.895 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:40:33.895 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:40:33.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:40:33.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:33.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:40:34.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T15:40:34.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T15:40:34.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T15:40:34.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:34.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:40:34.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-09T15:40:34.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-09T15:40:34.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672971' 2024-03-09T15:40:34.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:34.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:40:34.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-09T15:40:34.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-09T15:40:34.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672971 2-64424509447' 2024-03-09T15:40:34.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:34.291 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T15:40:34.291 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:34.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:40:34.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T15:40:34.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:34.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T15:40:34.293 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836502 2024-03-09T15:40:34.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T15:40:34.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:40:34.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T15:40:34.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:40:35.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:40:35.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:40:35.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-09T15:40:35.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:35.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-09T15:40:35.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:35.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:40:35.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-09T15:40:35.868 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:35.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-09T15:40:35.869 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672971 2024-03-09T15:40:35.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-09T15:40:35.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:40:36.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-03-09T15:40:36.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:36.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-09T15:40:36.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:36.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:40:36.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-09T15:40:36.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:36.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-09T15:40:36.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-09T15:40:36.144 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509447 2024-03-09T15:40:36.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:40:36.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-09T15:40:36.417 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:40:36.417 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:36.417 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:40:36.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:40:36.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:40:37.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:40:37.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:40:37.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:37.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:37.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:40:37.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:40:37.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:40:37.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-03-09T15:40:37.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-03-09T15:40:37.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:40:37.483 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:40:37.484 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:37.484 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:40:37.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:40:37.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:37.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:40:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:40:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:40:38.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:40:38.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:40:38.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:40:38.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:40:38.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:40:38.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:40:38.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:40:38.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:40:38.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:40:38.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T15:40:39.633 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:40:40.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:40:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:40:40.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:40:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:40:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:40:40.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:40:40.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:40:40.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:40:40.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:40:40.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:40:40.181 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:40.182+0000 7f275ba19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:40.184+0000 7f275ba19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:40.184 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:40.185+0000 7f275ba19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:40.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:40:40.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:40:40.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:40:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:40:40.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:40.706+0000 7f275ba19640 -1 Falling back to public interface 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:40:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:40:41.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:40:42.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:40:43.031 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:43.031+0000 7f275ba19640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:40:43.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:40:44.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:40:44.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:40:44.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:40:44.337 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:40:44.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:40:44.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:40:44.681 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3966194716,v1:127.0.0.1:6803/3966194716] [v2:127.0.0.1:6804/3966194716,v1:127.0.0.1:6805/3966194716] exists,up 508b1fba-ac18-4873-b896-220c97084676 2024-03-09T15:40:44.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:40:44.682 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:40:44.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:40:44.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:40:44.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:40:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:40:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:40:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:40:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:40:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:40:44.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:40:45.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:40:45.038 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:40:45.038 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:40:45.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:40:45.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:45.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:40:45.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T15:40:45.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T15:40:45.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T15:40:45.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:45.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:40:45.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T15:40:45.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T15:40:45.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-03-09T15:40:45.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:45.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:40:45.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T15:40:45.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T15:40:45.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509450' 2024-03-09T15:40:45.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:45.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T15:40:45.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:45.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:40:45.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T15:40:45.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:45.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T15:40:45.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T15:40:45.366 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182402 2024-03-09T15:40:45.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:40:45.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T15:40:45.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:45.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T15:40:45.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:45.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:40:45.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T15:40:45.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:45.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T15:40:45.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T15:40:45.671 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672975 2024-03-09T15:40:45.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:40:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T15:40:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:45.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T15:40:45.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:45.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:40:45.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T15:40:45.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:45.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T15:40:45.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T15:40:45.945 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509450 2024-03-09T15:40:45.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:40:46.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T15:40:46.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:40:46.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:46.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:46.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:40:46.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:40:46.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:40:46.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:40:46.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:40:46.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:40:46.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:40:46.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:40:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:40:46.827 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:40:46.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:46.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:47.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:40:47.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:40:47.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:40:47.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-09T15:40:47.258 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: { 2024-03-09T15:40:47.259 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T15:40:47.259 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:40:47.273 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: { 2024-03-09T15:40:47.273 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T15:40:47.273 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:40:47.286 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2: { 2024-03-09T15:40:47.287 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T15:40:47.287 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:40:47.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-03-09T15:40:47.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T15:40:47.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:40:47.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:47.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:40:47.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-03-09T15:40:47.603 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-03-09T15:40:47.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:40:47.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:40:47.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:47.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:40:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-03-09T15:40:47.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-03-09T15:40:47.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:40:47.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:40:47.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:40:47.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:40:48.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:48.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-03-09T15:40:48.245 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:40:48.245 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:40:48.245 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T15:40:48.245 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-02-24T15:39:08.246067+0000" 2024-03-09T15:40:48.245 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:40:48.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:48.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T15:40:48.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:48.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:40:48.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:40:48.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:40:48.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:40:26.781827+0000 '>' 2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:48.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:40:49.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:40:49.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:40:49.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:40:49.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:40:49.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:40:49.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:40:49.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:40:49.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:40:26.781827+0000 '>' 2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:49.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:40:50.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:40:50.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:40:51.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:40:26.781827+0000 '>' 2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:51.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:40:52.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:40:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:40:48.262462+0000 '>' 2024-03-09T15:40:26.781827+0000 2024-03-09T15:40:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:40:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:40:52.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:40:52.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:40:52.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:40:52.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:40:52.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:40:52.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:40:52.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:52.711 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:40:52.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-09T15:40:52.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-09T15:40:52.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-09T15:40:52.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:52.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:40:52.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T15:40:52.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T15:40:52.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672977' 2024-03-09T15:40:52.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:40:52.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:40:53.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T15:40:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T15:40:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672977 2-64424509453' 2024-03-09T15:40:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:53.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-09T15:40:53.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:53.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:40:53.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-09T15:40:53.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:53.053 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 107374182404 2024-03-09T15:40:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-09T15:40:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-09T15:40:53.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:40:53.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182404 2024-03-09T15:40:53.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:40:54.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:40:54.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:40:54.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182404 2024-03-09T15:40:54.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:54.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T15:40:54.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:54.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:40:54.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T15:40:54.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:54.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T15:40:54.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T15:40:54.611 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672977 2024-03-09T15:40:54.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:40:54.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T15:40:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:40:54.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T15:40:54.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:40:54.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:40:54.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T15:40:54.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:40:54.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T15:40:54.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T15:40:54.885 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509453 2024-03-09T15:40:54.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:40:55.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-09T15:40:55.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:40:55.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:55.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:55.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:40:55.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:40:55.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:40:55.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:40:55.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:40:55.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:40:55.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:40:55.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:40:55.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:40:55.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:40:55.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:40:55.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:40:56.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:40:56.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:40:56.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:40:56.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-03-09T15:40:56.372 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:40:56.513 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:40:48.428280+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:40:48.262462+0000 21'1 2024-03-09T15:40:48.262462+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:40:48.262462+0000 1 0 2024-03-09T15:40:56.514 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:40:56.514 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:40:56.515 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:40:56.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T15:40:56.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:40:56.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:40:56.515 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:40:56.516 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:40:56.516 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:40:56.516 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:56.516 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:40:56.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:40:56.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:40:56.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:40:56.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:40:56.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:40:56.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:40:56.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:40:56.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:40:56.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:40:57.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:40:57.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T15:40:58.402 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:40:58.402 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:40:58.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:40:58.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:40:58.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:40:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:40:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:40:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:40:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:40:58.700 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:58.699+0000 7f4ca9c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:58.702 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:58.701+0000 7f4ca9c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:58.703 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:58.703+0000 7f4ca9c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:40:58.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:40:58.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:40:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:40:59.222+0000 7f4ca9c48640 -1 Falling back to public interface 2024-03-09T15:40:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:00.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:00.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:00.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:41:00.226 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:41:00.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:00.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:01.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:01.455+0000 7f4ca9c48640 -1 osd.0 26 log_to_monitors true 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:01.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:01.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:02.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:03.058 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1278735865,v1:127.0.0.1:6803/1278735865] [v2:127.0.0.1:6804/1278735865,v1:127.0.0.1:6805/1278735865] exists,up 508b1fba-ac18-4873-b896-220c97084676 2024-03-09T15:41:03.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:41:03.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:41:03.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:41:03.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:03.411 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:41:03.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T15:41:03.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T15:41:03.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T15:41:03.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:03.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:41:03.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T15:41:03.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T15:41:03.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980' 2024-03-09T15:41:03.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:03.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:41:03.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T15:41:03.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T15:41:03.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980 2-64424509456' 2024-03-09T15:41:03.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:03.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T15:41:03.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:03.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:41:03.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T15:41:03.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:03.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T15:41:03.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T15:41:03.730 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 128849018882 2024-03-09T15:41:03.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:41:04.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:41:04.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:41:05.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:41:05.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:41:05.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:41:05.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:41:06.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:41:06.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:41:06.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T15:41:06.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:06.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T15:41:06.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:06.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:41:06.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T15:41:06.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:06.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T15:41:06.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T15:41:06.571 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672980 2024-03-09T15:41:06.571 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:41:06.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-03-09T15:41:06.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:06.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T15:41:06.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:06.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:41:06.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T15:41:06.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:06.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T15:41:06.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T15:41:06.844 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509456 2024-03-09T15:41:06.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:41:07.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T15:41:07.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:41:07.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:41:07.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:41:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:41:07.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:41:07.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:41:07.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:41:07.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:41:08.064 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:41:08.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:41:08.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:41:08.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:41:08.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:41:08.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:41:08.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:41:08.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:41:10.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T15:41:10.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:41:10.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:41:10.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:41:10.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:41:10.512 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:41:10.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:41:10.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:41:10.514 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:41:10.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:41:10.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:41:10.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:41:10.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:41:10.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:41:10.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:41:10.540 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:10.541+0000 7efc65a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:10.542 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:10.543+0000 7efc65a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:10.544 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:10.544+0000 7efc65a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:41:10.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:10.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:11.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:11.066 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:11.067+0000 7efc65a15640 -1 Falling back to public interface 2024-03-09T15:41:12.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:12.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:12.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:41:12.065 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:41:12.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:12.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:12.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:13.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:13.358+0000 7efc65a15640 -1 osd.0 31 log_to_monitors true 2024-03-09T15:41:13.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:14.969 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/79336374,v1:127.0.0.1:6803/79336374] [v2:127.0.0.1:6804/79336374,v1:127.0.0.1:6805/79336374] exists,up 508b1fba-ac18-4873-b896-220c97084676 2024-03-09T15:41:14.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:41:14.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:41:15.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:41:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:41:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:41:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:41:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:41:15.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:41:15.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:41:15.323 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:41:15.323 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:41:15.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:41:15.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:15.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:41:15.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T15:41:15.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T15:41:15.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-09T15:41:15.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:15.427 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:41:15.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-09T15:41:15.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-09T15:41:15.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672984' 2024-03-09T15:41:15.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:41:15.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:41:15.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T15:41:15.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T15:41:15.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672984 2-64424509459' 2024-03-09T15:41:15.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:15.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-09T15:41:15.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:15.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:41:15.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-09T15:41:15.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:15.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T15:41:15.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-09T15:41:15.638 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 150323855362 2024-03-09T15:41:15.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:41:15.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T15:41:15.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:15.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-09T15:41:15.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:15.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:41:15.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-09T15:41:15.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:15.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-09T15:41:15.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-09T15:41:15.935 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672984 2024-03-09T15:41:15.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:41:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672984 2024-03-09T15:41:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:41:17.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:41:17.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:41:17.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672984 2024-03-09T15:41:17.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:41:18.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:41:18.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:41:18.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-09T15:41:18.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:41:18.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-09T15:41:18.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:41:18.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:41:18.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-09T15:41:18.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:41:18.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T15:41:18.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-09T15:41:18.774 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509459 2024-03-09T15:41:18.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:41:19.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-09T15:41:19.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:41:19.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:41:19.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:41:19.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:41:19.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:41:19.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:41:19.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:41:19.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:41:19.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:41:19.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:41:19.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:41:19.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:41:19.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:41:19.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:41:19.990 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:40:48.261+0000 7f0c5fca3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:41:19.990 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:40:48.261+0000 7f0c5fca3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:41:19.990 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:40:48.261+0000 7f0c5fca3640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T15:41:19.990 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:40:48.261+0000 7f0c5fca3640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T15:41:19.990 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:40:48.261+0000 7f0c5fca3640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:41:19.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:41:19.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:41:19.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:41:20.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:41:20.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:41:20.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:41:20.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:41:20.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:41:20.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:41:20.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:41:20.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:41:20.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:41:20.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:41:20.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:41:20.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:41:20.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:41:20.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:41:20.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:41:20.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:41:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:41:20.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:41:20.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:41:20.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:41:20.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:41:20.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:41:20.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:41:20.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:41:20.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:41:20.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:41:20.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:41:20.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:41:20.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:41:20.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:41:20.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:41:20.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:41:20.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:41:20.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-03-09T15:41:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:41:20.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:41:20.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:41:20.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:41:20.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:41:20.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:41:20.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:41:20.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:41:20.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:41:20.513 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:41:20.513 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.513 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:20.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:41:20.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:41:20.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:41:20.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:41:20.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:41:20.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:41:20.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.708 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.709 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:41:20.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:41:20.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:20.787 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:41:20.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:41:20.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:41:20.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:41:21.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:41:21.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:41:21.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:41:21.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:41:21.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:41:21.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-03-09T15:41:21.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:41:21.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d8b44836-b922-4ac0-ab80-06b1886ff20a' 2024-03-09T15:41:21.708 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:41:21.708 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:41:21.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhg+xlGuAKKxAAnaTj6Hla8efQq+97syfKIw== 2024-03-09T15:41:21.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhg+xlGuAKKxAAnaTj6Hla8efQq+97syfKIw=="}' 2024-03-09T15:41:21.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8b44836-b922-4ac0-ab80-06b1886ff20a -i td/osd-scrub-repair/0/new.json 2024-03-09T15:41:21.928 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:41:21.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:41:21.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAhg+xlGuAKKxAAnaTj6Hla8efQq+97syfKIw== --osd-uuid d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:41:21.968 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.968+0000 7f42d0ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:21.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.978+0000 7f42d0ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:21.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.980+0000 7f42d0ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:21.980 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.980+0000 7f42d0ca2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:41:21.980 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.980+0000 7f42d0ca2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:41:21.980 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:21.980+0000 7f42d0ca2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:41:22.061 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:22.062+0000 7f42d0ca2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:41:27.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:41:27.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:41:27.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:41:27.560 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:41:27.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:41:27.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:41:27.906 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:41:27.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:41:27.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:41:27.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:41:27.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:41:27.934 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:27.935+0000 7f61bf98f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:27.936 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:27.937+0000 7f61bf98f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:27.937 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:27.938+0000 7f61bf98f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:28.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:28.177 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:41:28.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:41:28.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:28.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:28.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:28.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:28.457+0000 7f61bf98f640 -1 Falling back to public interface 2024-03-09T15:41:29.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:29.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:29.450 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:41:29.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:41:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:29.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:30.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:30.726 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:41:30.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:30.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:41:30.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:30.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:30.741 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:30.742+0000 7f61bf98f640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:41:31.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:32.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:32.160 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:41:32.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:32.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:41:32.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:32.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:33.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:33.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:33.474 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:41:33.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:41:33.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:33.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:33.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:34.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:34.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:34.757 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:41:34.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:41:34.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:34.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:35.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:36.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:36.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:41:36.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:41:36.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:36.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:36.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:37.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:37.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:37.330 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:41:37.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:41:37.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:37.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:37.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:38.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:38.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:38.635 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:41:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:41:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:38.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:39.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:39.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:39.922 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:41:39.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:41:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:40.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:41.207 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:41:41.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:41:41.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:41.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:41.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:42.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:42.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:42.989 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:41:42.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:41:42.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:42.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:43.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:44.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:44.573 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:41:44.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:44.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:41:44.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:44.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:44.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:45.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:45.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:45.999 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:41:45.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:41:45.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:45.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:46.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:47.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:47.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:47.591 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:41:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:41:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:48.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:49.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:49.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:49.115 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:41:49.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:41:49.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:49.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:49.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:50.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:50.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:50.876 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:41:50.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:41:50.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:50.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:51.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:41:52.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:41:52.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:52.476 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:41:52.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:41:52.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:52.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:41:52.849 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2298564002,v1:127.0.0.1:6803/2298564002] [v2:127.0.0.1:6804/2298564002,v1:127.0.0.1:6805/2298564002] exists,up d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:41:52.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:41:52.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:41:52.855 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:41:52.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 be9187c1-af98-4cb3-9f34-046812f52e7b' 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:41:52.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:41:52.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAg+xl9pHrMxAA979b+MBzoqZSazQMj6Zr3g== 2024-03-09T15:41:52.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAg+xl9pHrMxAA979b+MBzoqZSazQMj6Zr3g=="}' 2024-03-09T15:41:52.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be9187c1-af98-4cb3-9f34-046812f52e7b -i td/osd-scrub-repair/1/new.json 2024-03-09T15:41:53.288 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:41:53.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:41:53.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBAg+xl9pHrMxAA979b+MBzoqZSazQMj6Zr3g== --osd-uuid be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:41:53.330 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.331+0000 7eff31b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:53.372 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.373+0000 7eff31b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:53.373 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.374+0000 7eff31b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:53.374 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.374+0000 7eff31b45640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:41:53.374 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.374+0000 7eff31b45640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:41:53.374 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.374+0000 7eff31b45640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:41:53.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:53.444+0000 7eff31b45640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:41:59.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:41:59.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:41:59.204 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:41:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:41:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:41:59.556 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:41:59.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:41:59.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:41:59.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:41:59.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:41:59.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:41:59.584 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:59.585+0000 7f0f9f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:59.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:59.587+0000 7f0f9f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:59.587 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:41:59.588+0000 7f0f9f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:41:59.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:41:59.863 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:41:59.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:41:59.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:41:59.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:00.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:00.117+0000 7f0f9f7bb640 -1 Falling back to public interface 2024-03-09T15:42:00.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:01.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:01.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:01.132 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:42:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:42:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:01.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:01.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:02.414 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:42:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:42:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:02.468 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:02.469+0000 7f0f9f7bb640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:42:02.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:03.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:03.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:03.701 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:42:03.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:42:03.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:03.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:04.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:05.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:05.006 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:42:05.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:05.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:42:05.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:05.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:05.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:06.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:06.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:06.288 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:42:06.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:42:06.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:06.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:06.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:07.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:07.664 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:42:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:42:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:07.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:08.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:09.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:09.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:09.007 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:42:09.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:42:09.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:09.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:09.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:10.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:10.290 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:42:10.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:10.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:42:10.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:10.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:10.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:11.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:11.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:11.673 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:42:11.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:42:11.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:11.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:12.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:13.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:13.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:13.005 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:42:13.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:42:13.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:13.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:13.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:14.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:14.290 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:42:14.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:14.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:42:14.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:14.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:14.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:15.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:15.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:15.606 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:42:15.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:42:15.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:15.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:16.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:17.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:17.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:17.159 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:42:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:42:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:17.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:18.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:18.660 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:42:18.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:18.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:42:18.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:18.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:19.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:20.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:20.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:20.387 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:42:20.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:42:20.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:20.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:21.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:22.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:22.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:22.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:42:22.005 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:42:22.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:22.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:22.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:23.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:23.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:23.388 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:42:23.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:42:23.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:23.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:23.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:24.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:24.798 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:42:24.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:24.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:42:24.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:24.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:42:25.077 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4203453424,v1:127.0.0.1:6811/4203453424] [v2:127.0.0.1:6812/4203453424,v1:127.0.0.1:6813/4203453424] exists,up be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:42:25.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:42:25.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:42:25.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:42:25.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d763ca7-2d34-411b-a84c-f2e73ebcc0d8 2024-03-09T15:42:25.084 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 2d763ca7-2d34-411b-a84c-f2e73ebcc0d8 2024-03-09T15:42:25.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d763ca7-2d34-411b-a84c-f2e73ebcc0d8' 2024-03-09T15:42:25.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:42:25.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhg+xlHUTmBRAAlHnZpNQklS5Y7vNmNicUMQ== 2024-03-09T15:42:25.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhg+xlHUTmBRAAlHnZpNQklS5Y7vNmNicUMQ=="}' 2024-03-09T15:42:25.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d763ca7-2d34-411b-a84c-f2e73ebcc0d8 -i td/osd-scrub-repair/2/new.json 2024-03-09T15:42:25.457 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:42:25.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:42:25.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBhg+xlHUTmBRAAlHnZpNQklS5Y7vNmNicUMQ== --osd-uuid 2d763ca7-2d34-411b-a84c-f2e73ebcc0d8 2024-03-09T15:42:25.499 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.500+0000 7f353d087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:25.533 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.534+0000 7f353d087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:25.534 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.534+0000 7f353d087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:25.534 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.535+0000 7f353d087640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:42:25.535 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.535+0000 7f353d087640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:42:25.535 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.535+0000 7f353d087640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:42:25.611 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:25.611+0000 7f353d087640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:42:31.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:42:31.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:42:31.534 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:42:31.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:42:31.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:42:31.903 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:42:31.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:42:31.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:42:31.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:42:31.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:42:31.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:42:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:31.931+0000 7f10a0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:31.932 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:31.933+0000 7f10a0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:31.934+0000 7f10a0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:42:32.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:42:32.176 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:42:32.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:32.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:32.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:32.463 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:32.464+0000 7f10a0c0b640 -1 Falling back to public interface 2024-03-09T15:42:33.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:33.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:33.448 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:42:33.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:42:33.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:33.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:33.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:34.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:34.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:34.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:42:34.727 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:42:34.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:34.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:34.791 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:42:34.791+0000 7f10a0c0b640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:42:35.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:36.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:36.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:36.008 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:42:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:42:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:37.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:37.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:37.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:42:37.290 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:42:37.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:37.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:37.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:38.637 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:42:38.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:42:38.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:38.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:38.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:39.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:39.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:39.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:42:39.920 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:42:39.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:39.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:41.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:41.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:41.202 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:42:41.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:42:41.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:41.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:41.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:42.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:42.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:42.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:42:42.570 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:42:42.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:42.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:42.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:43.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:43.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:43.853 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:42:43.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:42:43.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:43.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:44.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:45.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:45.139 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:42:45.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:42:45.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:45.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:45.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:46.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:46.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:46.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:42:46.425 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:42:46.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:46.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:46.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:47.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:47.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:47.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:42:47.731 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:42:47.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:47.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:48.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:49.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:49.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:49.073 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:42:49.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:42:49.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:49.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:49.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:50.579 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:42:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:42:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:50.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:51.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:52.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:52.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:52.022 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:42:52.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:42:52.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:52.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:52.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:53.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:53.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:53.571 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:42:53.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:42:53.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:53.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:53.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:54.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:54.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:54.957 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:42:54.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:42:54.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:54.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:55.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:56.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:56.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:56.444 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:42:56.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:42:56.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:56.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:56.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:42:57.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:42:57.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:42:57.818 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:42:57.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:42:57.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:42:57.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:42:58.097 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1616341205,v1:127.0.0.1:6819/1616341205] [v2:127.0.0.1:6820/1616341205,v1:127.0.0.1:6821/1616341205] exists,up 2d763ca7-2d34-411b-a84c-f2e73ebcc0d8 2024-03-09T15:42:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:42:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:42:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:42:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-03-09T15:42:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T15:42:58.497 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T15:42:58.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:42:59.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-03-09T15:43:00.076 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T15:43:00.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-09T15:43:00.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:43:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:43:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:43:00.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:43:00.089 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:43:00.089 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:43:00.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:43:00.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:43:00.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:43:00.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:43:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:43:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:43:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:43:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:00.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:00.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:00.516 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:00.516 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:00.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:00.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:00.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:00.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T15:43:00.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T15:43:00.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T15:43:00.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:00.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:00.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T15:43:00.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T15:43:00.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672973' 2024-03-09T15:43:00.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:00.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672973 2-64424509445' 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T15:43:00.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:00.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:00.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T15:43:00.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:00.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T15:43:00.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T15:43:00.933 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836498 2024-03-09T15:43:00.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-09T15:43:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:02.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:02.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:02.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T15:43:02.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:02.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T15:43:02.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:02.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:02.565 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T15:43:02.565 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:02.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T15:43:02.566 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672973 2024-03-09T15:43:02.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T15:43:02.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:02.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T15:43:02.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:02.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T15:43:02.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:02.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:02.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T15:43:02.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T15:43:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T15:43:02.840 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509445 2024-03-09T15:43:02.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:03.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-09T15:43:03.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:43:03.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:03.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:03.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:03.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:43:03.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:03.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:03.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:04.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:43:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T15:43:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:43:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T15:43:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:43:04.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:04.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:04.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:43:04.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:04.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:04.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:04.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:43:04.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T15:43:04.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:43:04.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:43:04.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:43:04.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:43:04.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:43:04.762 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:43:04.762 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:43:05.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:43:05.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:43:05.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T15:43:05.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:43:05.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T15:43:05.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:43:05.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:05.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:05.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:05.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:05.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:05.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:05.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:05.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:43:05.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:05.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:05.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:43:05.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:43:05.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:43:05.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:43:06.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:43:06.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:43:06.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T15:43:06.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:43:06.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:06.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:06.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:43:06.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:06.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:06.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-03-09T15:43:07.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-03-09T15:43:07.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:07.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:07.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:07.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:07.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:07.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:07.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:08.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:08.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:08.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:08.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:08.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:08.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:08.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T15:43:08.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:43:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:43:08.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:43:08.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:43:08.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:43:09.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:43:09.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:43:09.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:09.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-09T15:43:10.647 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:43:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:43:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:43:11.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:43:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:43:11.201 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:11.202+0000 7f01cfb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:11.205+0000 7f01cfb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:11.205 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:11.206+0000 7f01cfb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:11.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:11.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:11.728 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:11.729+0000 7f01cfb71640 -1 Falling back to public interface 2024-03-09T15:43:12.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:12.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:12.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:43:12.726 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:12.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:12.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:13.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:14.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:14.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:14.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:43:14.007 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:43:14.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:14.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:14.212 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:14.212+0000 7f01cfb71640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:43:14.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:15.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:15.600 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3675301484,v1:127.0.0.1:6803/3675301484] [v2:127.0.0.1:6804/3675301484,v1:127.0.0.1:6805/3675301484] exists,up d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:43:15.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:43:15.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:43:15.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:43:15.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:43:15.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:43:15.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:15.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:15.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T15:43:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T15:43:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T15:43:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:16.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:16.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T15:43:16.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T15:43:16.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-03-09T15:43:16.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:16.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509449' 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T15:43:16.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:16.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:16.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T15:43:16.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:16.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T15:43:16.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T15:43:16.323 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182402 2024-03-09T15:43:16.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T15:43:16.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:17.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:17.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:17.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T15:43:17.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:17.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T15:43:17.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:17.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:17.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T15:43:17.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:17.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T15:43:17.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T15:43:17.954 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672977 2024-03-09T15:43:17.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T15:43:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-09T15:43:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:18.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-09T15:43:18.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-09T15:43:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-09T15:43:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509449 2024-03-09T15:43:18.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:18.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-09T15:43:18.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:43:18.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:18.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:18.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:18.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:19.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:43:19.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:19.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:19.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:43:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:43:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:43:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:43:19.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:43:19.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:43:19.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:43:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:43:20.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:21.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T15:43:22.425 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T15:43:22.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:22.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:43:22.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:43:22.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:43:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:43:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:43:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:43:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:43:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:43:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:43:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:43:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:43:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:43:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:43:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:43:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:22.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:43:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:43:22.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:43:22.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:43:22.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:43:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:22.975+0000 7f05325fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:22.977+0000 7f05325fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:22.979+0000 7f05325fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:23.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:43:23.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:23.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:23.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:23.496 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:23.496+0000 7f05325fd640 -1 Falling back to public interface 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:24.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:24.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:25.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:25.830 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:25.829+0000 7f05325fd640 -1 osd.0 26 log_to_monitors true 2024-03-09T15:43:26.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:27.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:27.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:27.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:43:27.103 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:43:27.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:27.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:43:27.383 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/4097217270,v1:127.0.0.1:6803/4097217270] [v2:127.0.0.1:6804/4097217270,v1:127.0.0.1:6805/4097217270] exists,up d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:43:27.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:43:27.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:43:27.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:43:27.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:43:27.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:43:27.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:43:27.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:43:27.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:43:27.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:43:27.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:27.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:27.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:27.738 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:27.738 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:27.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:27.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:27.739 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T15:43:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T15:43:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T15:43:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:27.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:27.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T15:43:27.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T15:43:27.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980' 2024-03-09T15:43:27.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:27.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:28.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-09T15:43:28.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-09T15:43:28.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672980 2-64424509452' 2024-03-09T15:43:28.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:28.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T15:43:28.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:28.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:28.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T15:43:28.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:28.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T15:43:28.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T15:43:28.055 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 128849018882 2024-03-09T15:43:28.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:28.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:43:28.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:29.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:29.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:29.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:43:29.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:30.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:43:30.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:30.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T15:43:30.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:30.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T15:43:30.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:30.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:30.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T15:43:30.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:30.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T15:43:30.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T15:43:30.879 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672980 2024-03-09T15:43:30.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:31.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-09T15:43:31.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:31.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-09T15:43:31.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:31.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:31.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-09T15:43:31.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:31.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-09T15:43:31.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-09T15:43:31.154 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509452 2024-03-09T15:43:31.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:31.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-03-09T15:43:31.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:43:31.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:31.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:31.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:32.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:43:32.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:32.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:32.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:32.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:43:32.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:43:32.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:43:32.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:43:32.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:43:32.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:43:32.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:43:32.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:43:32.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:43:32.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:43:32.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:43:32.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-09T15:43:34.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:34.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:43:34.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:43:34.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:43:34.845 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:43:34.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:43:34.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:43:34.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:43:34.847 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:43:34.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:34.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:43:34.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:43:34.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:43:34.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:43:34.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:43:34.876 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:34.877+0000 7fdcc9338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:34.879 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:34.879+0000 7fdcc9338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:34.880 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:34.881+0000 7fdcc9338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:35.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:43:35.127 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:43:35.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:35.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:43:35.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:35.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:35.403+0000 7fdcc9338640 -1 Falling back to public interface 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:43:36.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:37.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:37.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:37.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:43:37.671 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:43:37.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:37.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:43:37.723 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:37.723+0000 7fdcc9338640 -1 osd.1 31 log_to_monitors true 2024-03-09T15:43:37.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:43:38.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:43:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:43:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:43:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:43:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:43:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:43:39.240 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/3701092801,v1:127.0.0.1:6811/3701092801] [v2:127.0.0.1:6812/3701092801,v1:127.0.0.1:6813/3701092801] exists,up be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:43:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:39.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:39.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-09T15:43:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-09T15:43:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-09T15:43:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:39.698 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:39.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T15:43:39.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T15:43:39.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-03-09T15:43:39.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:39.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T15:43:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T15:43:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509456' 2024-03-09T15:43:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:39.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-09T15:43:39.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:39.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:39.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-09T15:43:39.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:39.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-09T15:43:39.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-09T15:43:39.927 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 128849018885 2024-03-09T15:43:39.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:40.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-09T15:43:40.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:40.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-09T15:43:40.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:40.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:40.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-09T15:43:40.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T15:43:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-09T15:43:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 150323855362 2024-03-09T15:43:40.197 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:40.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-09T15:43:40.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:41.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:41.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:41.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-09T15:43:41.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:42.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:43:42.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:43.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T15:43:43.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:43.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T15:43:43.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:43.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:43.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T15:43:43.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:43.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T15:43:43.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T15:43:43.035 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509456 2024-03-09T15:43:43.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:43.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T15:43:43.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:43:43.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:43.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:43.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:43.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:43:43.958 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:43.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:43.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:44.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:43:44.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T15:43:44.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:43:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-03-09T15:43:44.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-03-09T15:43:44.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:43:44.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T15:43:44.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T15:43:44.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:43:44.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-03-09T15:43:44.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-03-09T15:43:44.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:44.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:44.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:44.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:45.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:45.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-03-09T15:43:45.233 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:43:45.233 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:43:45.233 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T15:43:45.233 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-02-24T15:42:05.234709+0000" 2024-03-09T15:43:45.234 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:45.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:45.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:42:58.450153+0000 '>' 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:45.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:43:46.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:43:46.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:43:46.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:43:46.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:46.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:46.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:46.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:46.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:42:58.450153+0000 '>' 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:46.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:43:47.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:43:47.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:43:47.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:43:47.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:47.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:47.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:47.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:48.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:42:58.450153+0000 '>' 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:48.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:43:49.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:43:49.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:43:49.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:43:49.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:49.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:49.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:49.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:42:58.450153+0000 '>' 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:43:50.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:43:50.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:43:50.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:43:50.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:43:50.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:43:50.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:43:50.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:43:46.309925+0000 '>' 2024-03-09T15:42:58.450153+0000 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:43:50.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:50.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:51.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:51.016 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:51.016 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:51.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:51.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:51.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:51.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018889 2024-03-09T15:43:51.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018889 2024-03-09T15:43:51.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889' 2024-03-09T15:43:51.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:51.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:51.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-09T15:43:51.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-09T15:43:51.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365' 2024-03-09T15:43:51.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:51.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:51.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T15:43:51.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T15:43:51.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365 2-64424509459' 2024-03-09T15:43:51.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:51.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018889 2024-03-09T15:43:51.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:51.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:51.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018889 2024-03-09T15:43:51.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:51.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018889 2024-03-09T15:43:51.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018889' 2024-03-09T15:43:51.336 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 128849018889 2024-03-09T15:43:51.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:51.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018889 2024-03-09T15:43:51.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:52.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:52.602 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:52.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018889 2024-03-09T15:43:52.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:52.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-03-09T15:43:52.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:52.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:52.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-03-09T15:43:52.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:52.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-09T15:43:52.885 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 150323855365 2024-03-09T15:43:52.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-03-09T15:43:52.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:53.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-09T15:43:53.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:53.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-09T15:43:53.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:53.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:53.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-09T15:43:53.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:53.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T15:43:53.160 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509459 2024-03-09T15:43:53.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-09T15:43:53.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:53.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-09T15:43:53.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:43:53.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:53.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:43:53.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:43:54.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:43:54.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:43:54.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:43:54.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:43:54.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:43:54.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:43:54.650 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:43:54.650 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:43:54.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:43:54.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:54.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:43:54.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-03-09T15:43:54.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-03-09T15:43:54.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-03-09T15:43:54.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:54.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:43:54.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-03-09T15:43:54.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-03-09T15:43:54.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-03-09T15:43:54.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:43:54.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:43:54.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-09T15:43:54.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-09T15:43:54.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509461' 2024-03-09T15:43:54.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:54.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-03-09T15:43:54.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:54.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:43:54.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-03-09T15:43:54.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:54.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-03-09T15:43:54.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-03-09T15:43:54.964 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 128849018890 2024-03-09T15:43:54.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:55.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-03-09T15:43:55.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:43:56.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:43:56.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:43:56.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-03-09T15:43:56.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:56.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-03-09T15:43:56.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:56.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:43:56.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-03-09T15:43:56.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:56.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-03-09T15:43:56.519 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 150323855367 2024-03-09T15:43:56.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-03-09T15:43:56.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:43:56.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-03-09T15:43:56.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:43:56.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-09T15:43:56.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:43:56.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:43:56.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-09T15:43:56.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:43:56.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-09T15:43:56.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-09T15:43:56.794 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509461 2024-03-09T15:43:56.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:43:57.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-09T15:43:57.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:43:57.069 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T15:43:57.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-03-09T15:43:57.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T15:43:57.327 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:43:57.327 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-09T15:43:46.310018+0000 21'10 36:100 [1,0] 1 [1,0] 1 21'10 2024-03-09T15:43:46.309925+0000 21'10 2024-03-09T15:43:46.309925+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:43:46.309925+0000 10 0 2024-03-09T15:43:57.327 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:43:57.327 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:43:57.327 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:43:57.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T15:43:57.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-03-09T15:43:57.591 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T15:43:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:43:57.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:43:57.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T15:43:57.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:43:58.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T15:43:58.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:43:58.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:43:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:58.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-03-09T15:43:59.614 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:43:59.615 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:43:59.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:59.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:43:59.892 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:43:59.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:43:59.894 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:43:59.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:43:59.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:43:59.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:43:59.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:43:59.921 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:59.922+0000 7f0f23c9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:59.935 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:59.936+0000 7f0f23c9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:43:59.937 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:43:59.937+0000 7f0f23c9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:44:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:44:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:44:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:44:00.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:44:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:44:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:44:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:00.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:00.461 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:00.462+0000 7f0f23c9d640 -1 Falling back to public interface 2024-03-09T15:44:01.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:44:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:44:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:01.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:01.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:02.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:02.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:02.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:44:02.727 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:44:02.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:02.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:02.768 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:02.768+0000 7f0f23c9d640 -1 osd.0 36 log_to_monitors true 2024-03-09T15:44:03.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:04.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:44:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:44:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/1466035382,v1:127.0.0.1:6803/1466035382] [v2:127.0.0.1:6804/1466035382,v1:127.0.0.1:6805/1466035382] exists,up d8b44836-b922-4ac0-ab80-06b1886ff20a 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:44:04.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:44:04.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:44:04.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:44:04.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:44:04.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:44:04.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:44:04.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:44:04.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:44:04.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:44:04.811 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:44:04.811 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:44:04.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:44:04.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:04.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:44:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T15:44:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T15:44:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-03-09T15:44:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:04.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:44:05.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-03-09T15:44:05.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-03-09T15:44:05.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-03-09T15:44:05.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:05.018 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:44:05.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-09T15:44:05.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-09T15:44:05.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509464' 2024-03-09T15:44:05.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:05.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-03-09T15:44:05.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:05.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:44:05.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-03-09T15:44:05.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:05.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T15:44:05.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-03-09T15:44:05.124 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 171798691842 2024-03-09T15:44:05.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:44:05.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-03-09T15:44:05.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:44:06.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:44:06.396 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:44:06.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T15:44:06.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:06.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-03-09T15:44:06.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:06.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:44:06.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-03-09T15:44:06.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:06.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-03-09T15:44:06.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-03-09T15:44:06.678 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 150323855370 2024-03-09T15:44:06.678 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:44:06.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-03-09T15:44:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:06.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-09T15:44:06.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:44:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-09T15:44:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:06.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-09T15:44:06.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-09T15:44:06.953 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509464 2024-03-09T15:44:06.953 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:44:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-09T15:44:07.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:44:07.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:44:07.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:44:07.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:44:07.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:44:07.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:44:07.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:44:07.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:44:07.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:44:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:44:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:44:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:44:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-03-09T15:44:08.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:44:08.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:44:08.203 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:44:08.204 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T15:44:08.204 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:44:08.204 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T15:44:08.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-09T15:44:08.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:44:08.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:44:08.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:44:08.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:44:08.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-03-09T15:44:08.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:44:08.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:44:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:44:08.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:44:08.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-03-09T15:44:10.119 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-03-09T15:44:10.119 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:44:10.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:10.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:44:10.639 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:44:10.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:44:10.640 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:44:10.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:44:10.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:44:10.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:44:10.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:44:10.669 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:10.669+0000 7f0cdba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:10.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:10.672+0000 7f0cdba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:10.672 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:10.673+0000 7f0cdba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:10.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:44:10.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:44:10.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:10.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:44:11.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:11.190 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:11.191+0000 7f0cdba0f640 -1 Falling back to public interface 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:12.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:44:12.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:13.473 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:13.474+0000 7f0cdba0f640 -1 osd.1 41 log_to_monitors true 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:13.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:44:13.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:14.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:14.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:14.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:44:14.760 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:44:14.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:14.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/1037685629,v1:127.0.0.1:6811/1037685629] [v2:127.0.0.1:6812/1037685629,v1:127.0.0.1:6813/1037685629] exists,up be9187c1-af98-4cb3-9f34-046812f52e7b 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:44:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:44:15.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:44:15.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:44:15.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:44:15.398 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:44:15.398 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:44:15.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:44:15.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:15.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:44:15.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T15:44:15.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T15:44:15.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-09T15:44:15.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:15.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:44:15.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-03-09T15:44:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-03-09T15:44:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-03-09T15:44:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:15.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509467' 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-09T15:44:15.708 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:15.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:44:15.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-09T15:44:15.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:15.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T15:44:15.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-09T15:44:15.714 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 171798691845 2024-03-09T15:44:15.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:44:15.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-09T15:44:15.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:15.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-03-09T15:44:15.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:15.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:44:15.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-03-09T15:44:15.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:15.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-03-09T15:44:15.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-03-09T15:44:15.997 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 193273528322 2024-03-09T15:44:15.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:44:16.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-03-09T15:44:16.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:16.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-09T15:44:16.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:16.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:44:16.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-09T15:44:16.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-09T15:44:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-09T15:44:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509467 2024-03-09T15:44:16.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:44:16.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-09T15:44:16.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:44:16.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:44:16.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:44:16.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:44:17.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:44:17.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:44:17.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:44:17.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:44:17.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:44:17.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:44:17.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:44:17.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:44:17.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:44:17.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:44:17.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T15:44:17.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:44:17.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:44:17.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:44:17.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:44:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T15:44:18.041 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:44:18.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:44:18.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:44:18.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:44:18.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:44:18.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:44:18.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:44:18.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:43:46.309925+0000 '>' 2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:18.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:44:19.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:44:19.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:44:19.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:44:19.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:44:19.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:44:19.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:44:19.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:44:19.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:43:46.309925+0000 '>' 2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:19.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:44:20.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:44:20.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:44:20.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:44:20.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:44:20.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:44:20.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:44:20.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:44:20.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:44:18.400650+0000 '>' 2024-03-09T15:43:46.309925+0000 2024-03-09T15:44:20.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:44:20.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-03-09T15:44:22.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-09T15:44:22.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:44:22.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:44:23.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:44:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:44:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:44:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:44:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:23.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:44:23.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-03-09T15:44:23.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-03-09T15:44:23.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848' 2024-03-09T15:44:23.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:23.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:44:23.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528324 2024-03-09T15:44:23.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528324 2024-03-09T15:44:23.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528324' 2024-03-09T15:44:23.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:44:23.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:44:23.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528324 2-64424509470' 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691848 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:23.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:44:23.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691848 2024-03-09T15:44:23.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:23.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-03-09T15:44:23.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691848' 2024-03-09T15:44:23.470 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 171798691848 2024-03-09T15:44:23.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:44:23.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691848 2024-03-09T15:44:23.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:44:24.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:44:24.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:44:25.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-03-09T15:44:25.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:25.025 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528324 2024-03-09T15:44:25.025 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:25.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:44:25.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528324 2024-03-09T15:44:25.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:25.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528324 2024-03-09T15:44:25.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528324' 2024-03-09T15:44:25.028 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 193273528324 2024-03-09T15:44:25.028 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:44:25.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528324 2024-03-09T15:44:25.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:44:25.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-03-09T15:44:25.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:44:25.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:44:25.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-03-09T15:44:25.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:44:25.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-03-09T15:44:25.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-03-09T15:44:25.303 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509470 2024-03-09T15:44:25.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:44:25.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-03-09T15:44:25.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T15:44:25.832 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:44:25.832 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-03-09T15:44:18.400748+0000 21'10 46:138 [1,0] 1 [1,0] 1 21'10 2024-03-09T15:44:18.400650+0000 21'10 2024-03-09T15:44:18.400650+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:44:18.400650+0000 9 0 2024-03-09T15:44:25.832 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:44:25.832 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:44:25.832 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:44:25.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T15:44:25.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-03-09T15:44:26.096 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:44:26.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:44:26.109 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:44:26.109 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.097+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:43:46.309+0000 7fdcad2b1640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:44:18.399+0000 7f0cbf392640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:44:18.399+0000 7f0cbf392640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:44:18.399+0000 7f0cbf392640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:44:18.399+0000 7f0cbf392640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-03-09T15:44:26.110 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:44:18.399+0000 7f0cbf392640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:44:26.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:44:26.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:44:26.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:44:26.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:44:26.218 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:44:26.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:44:26.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:44:26.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:44:26.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:44:26.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:44:26.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:44:26.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:44:26.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:44:26.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:44:26.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:44:26.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:44:26.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:44:26.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:44:26.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:44:26.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:44:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:44:26.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:44:26.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:44:26.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:44:26.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:44:26.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:44:26.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:44:26.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:44:26.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:44:26.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:44:26.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:44:26.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:44:26.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:44:26.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:44:26.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:44:26.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:44:26.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:44:26.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:44:26.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:44:26.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:44:26.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:44:26.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-03-09T15:44:26.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:44:26.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:26.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:44:26.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:44:26.807 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:44:26.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:44:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:44:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:44:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:44:26.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:44:26.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:44:26.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:27.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:27.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:44:27.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:44:27.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:44:27.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-03-09T15:44:27.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T15:44:27.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:44:27.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:44:27.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:44:27.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:44:27.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:44:27.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:44:27.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:44:27.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6' 2024-03-09T15:44:27.812 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:44:27.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:44:27.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbg+xluadYMRAAG803ymrDu5Xvqts4PWBKDA== 2024-03-09T15:44:27.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbg+xluadYMRAAG803ymrDu5Xvqts4PWBKDA=="}' 2024-03-09T15:44:27.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 -i td/osd-scrub-repair/0/new.json 2024-03-09T15:44:28.030 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:44:28.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:44:28.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDbg+xluadYMRAAG803ymrDu5Xvqts4PWBKDA== --osd-uuid 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:44:28.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.074+0000 7f60ee40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:28.080 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.080+0000 7f60ee40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:28.081 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.081+0000 7f60ee40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:28.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.081+0000 7f60ee40d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:44:28.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.081+0000 7f60ee40d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:44:28.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.081+0000 7f60ee40d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:44:28.164 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:28.163+0000 7f60ee40d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:44:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:44:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:44:33.825 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:44:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:44:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:44:34.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:44:34.179 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:44:34.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:44:34.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:44:34.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:44:34.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:44:34.207 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:34.206+0000 7fa8c6f91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:34.208 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:34.208+0000 7fa8c6f91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:34.209 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:34.209+0000 7fa8c6f91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:34.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:44:34.455 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:44:34.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:34.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:34.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:34.728 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:34.728+0000 7fa8c6f91640 -1 Falling back to public interface 2024-03-09T15:44:35.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:35.727 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:44:35.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:35.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:44:35.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:35.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:36.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:37.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:37.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:37.007 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:44:37.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:44:37.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:37.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:37.053 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:44:37.053+0000 7fa8c6f91640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:44:37.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:38.338 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:44:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:44:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:38.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:39.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:39.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:39.621 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:44:39.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:44:39.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:39.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:40.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:41.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:41.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:41.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:44:41.024 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:44:41.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:41.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:41.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:42.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:42.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:42.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:44:42.306 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:44:42.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:42.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:42.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:43.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:43.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:43.623 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:44:43.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:44:43.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:43.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:43.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:44.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:44.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:44.910 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:44:44.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:44:44.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:44.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:45.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:46.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:46.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:46.190 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:44:46.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:44:46.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:46.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:46.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:47.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:47.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:47.472 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:44:47.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:44:47.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:47.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:47.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:48.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:48.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:48.995 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:44:48.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:44:48.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:48.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:49.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:50.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:50.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:50.278 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:44:50.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:44:50.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:50.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:50.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:51.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:51.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:51.689 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:44:51.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:44:51.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:51.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:52.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:53.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:53.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:53.332 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:44:53.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:44:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:54.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:55.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:55.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:55.210 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:44:55.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:44:55.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:55.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:55.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:56.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:56.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:56.780 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:44:56.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:44:56.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:56.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:58.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:58.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:58.113 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:44:58.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:44:58.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:58.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:58.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:44:59.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:44:59.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:44:59.512 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:44:59.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:44:59.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:44:59.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:44:59.788 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4193063848,v1:127.0.0.1:6803/4193063848] [v2:127.0.0.1:6804/4193063848,v1:127.0.0.1:6805/4193063848] exists,up 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:44:59.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:44:59.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:44:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:44:59.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:44:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:44:59.796 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:44:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89d0e760-5d49-49fa-b097-d90b135fe4f1' 2024-03-09T15:44:59.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:44:59.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7g+xlkP1PMBAAPkwJjez0a6MiI32Wj9TtkA== 2024-03-09T15:44:59.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7g+xlkP1PMBAAPkwJjez0a6MiI32Wj9TtkA=="}' 2024-03-09T15:44:59.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89d0e760-5d49-49fa-b097-d90b135fe4f1 -i td/osd-scrub-repair/1/new.json 2024-03-09T15:45:00.183 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:45:00.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:45:00.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD7g+xlkP1PMBAAPkwJjez0a6MiI32Wj9TtkA== --osd-uuid 89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:45:00.225 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.225+0000 7f82905e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:00.267 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.268+0000 7f82905e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.269+0000 7f82905e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.269+0000 7f82905e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:45:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.269+0000 7f82905e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:45:00.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.269+0000 7f82905e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:45:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:00.351+0000 7f82905e1640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:45:06.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:45:06.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:45:06.308 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:45:06.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:45:06.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:45:06.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:45:06.658 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:45:06.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:45:06.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:45:06.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:45:06.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:45:06.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:06.687+0000 7fe932c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:06.688 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:06.689+0000 7fe932c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:06.689 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:06.690+0000 7fe932c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:45:06.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:06.944 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:45:06.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:45:06.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:06.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:07.202 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:07.203+0000 7fe932c0d640 -1 Falling back to public interface 2024-03-09T15:45:07.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:08.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:08.219 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:45:08.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:08.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:45:08.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:08.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:08.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:09.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:09.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:09.501 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:45:09.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:45:09.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:09.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:09.555 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:09.555+0000 7fe932c0d640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:45:09.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:10.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:10.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:10.778 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:45:10.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:45:10.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:10.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:11.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:12.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:12.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:12.141 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:45:12.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:45:12.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:12.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:12.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:13.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:13.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:13.537 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:45:13.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:45:13.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:13.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:13.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:14.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:14.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:14.818 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:45:14.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:45:14.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:14.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:15.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:16.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:16.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:16.146 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:45:16.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:45:16.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:16.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:16.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:17.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:17.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:17.527 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:45:17.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:45:17.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:17.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:17.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:18.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:18.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:18.810 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:45:18.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:45:18.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:18.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:19.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:20.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:20.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:20.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:45:20.145 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:45:20.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:20.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:20.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:21.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:21.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:21.428 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:45:21.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:45:21.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:21.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:21.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:22.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:22.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:22.876 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:45:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:45:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:23.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:24.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:24.537 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:45:24.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:24.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:45:24.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:24.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:25.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:26.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:26.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:26.143 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:45:26.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:45:26.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:26.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:26.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:27.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:27.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:27.507 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:45:27.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:45:27.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:27.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:28.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:29.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:29.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:29.060 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:45:29.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:45:29.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:29.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:29.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:30.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:30.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:30.887 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:45:30.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:45:30.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:30.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:31.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:32.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:32.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:32.606 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:45:32.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:45:32.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:32.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:45:32.927 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2836110204,v1:127.0.0.1:6811/2836110204] [v2:127.0.0.1:6812/2836110204,v1:127.0.0.1:6813/2836110204] exists,up 89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:45:32.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:45:32.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:45:32.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:45:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:45:32.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:45:32.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=876661f0-4614-44a0-98a3-d5f7726a0dec 2024-03-09T15:45:32.935 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 876661f0-4614-44a0-98a3-d5f7726a0dec 2024-03-09T15:45:32.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 876661f0-4614-44a0-98a3-d5f7726a0dec' 2024-03-09T15:45:32.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:45:32.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAchOxlkyOOOBAALNbfVSu6vwrpemLo19vNcg== 2024-03-09T15:45:32.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAchOxlkyOOOBAALNbfVSu6vwrpemLo19vNcg=="}' 2024-03-09T15:45:32.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 876661f0-4614-44a0-98a3-d5f7726a0dec -i td/osd-scrub-repair/2/new.json 2024-03-09T15:45:33.349 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:45:33.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:45:33.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAchOxlkyOOOBAALNbfVSu6vwrpemLo19vNcg== --osd-uuid 876661f0-4614-44a0-98a3-d5f7726a0dec 2024-03-09T15:45:33.391 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.390+0000 7f9e3be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:33.433 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.433+0000 7f9e3be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:33.434 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.434+0000 7f9e3be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:33.434 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.434+0000 7f9e3be0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:45:33.435 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.434+0000 7f9e3be0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:45:33.435 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.434+0000 7f9e3be0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:45:33.508 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:33.508+0000 7f9e3be0e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:45:39.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:45:39.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:45:39.423 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:45:39.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:45:39.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:45:39.774 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:45:39.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:45:39.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:45:39.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:45:39.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:45:39.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:45:39.803 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:39.802+0000 7f4273d9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:39.805 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:39.804+0000 7f4273d9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:39.805 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:39.805+0000 7f4273d9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:40.047 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:45:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:45:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:40.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:40.324 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:40.324+0000 7f4273d9a640 -1 Falling back to public interface 2024-03-09T15:45:41.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:41.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:41.319 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:45:41.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:45:41.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:41.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:41.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:42.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:42.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:42.601 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:45:42.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:45:42.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:42.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:42.697 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:45:42.696+0000 7f4273d9a640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:45:42.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:43.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:43.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:43.898 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:45:43.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:45:43.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:43.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:44.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:45.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:45.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:45.182 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:45:45.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:45:45.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:45.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:45.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:46.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:46.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:46.513 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:45:46.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:45:46.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:46.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:46.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:47.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:47.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:47.793 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:45:47.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:45:47.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:47.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:48.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:49.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:49.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:49.078 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:45:49.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:45:49.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:49.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:50.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:50.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:50.361 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:45:50.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:45:50.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:50.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:50.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:51.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:51.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:51.711 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:45:51.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:45:51.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:51.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:51.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:52.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:52.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:52.997 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:45:52.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:45:52.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:52.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:53.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:54.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:54.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:54.355 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:45:54.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:45:54.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:54.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:55.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:56.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:56.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:56.129 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:45:56.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:45:56.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:56.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:57.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:58.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:58.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:58.046 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:45:58.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:45:58.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:58.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:45:58.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:45:59.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:45:59.798 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:45:59.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:45:59.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:45:59.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:45:59.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:46:00.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:01.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:01.200 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:46:01.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:46:01.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:01.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:46:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:02.944 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:46:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:46:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:46:03.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:04.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:04.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:04.536 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:46:04.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:46:04.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:04.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:46:04.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:05.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:05.957 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:46:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:46:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:05.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:46:06.238 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3000356432,v1:127.0.0.1:6819/3000356432] [v2:127.0.0.1:6820/3000356432,v1:127.0.0.1:6821/3000356432] exists,up 876661f0-4614-44a0-98a3-d5f7726a0dec 2024-03-09T15:46:06.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:46:06.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:46:06.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:46:06.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-03-09T15:46:06.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T15:46:06.604 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T15:46:06.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:46:07.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-03-09T15:46:08.126 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T15:46:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-09T15:46:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:46:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:46:08.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:46:08.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:46:08.139 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:46:08.139 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:46:08.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:46:08.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:46:08.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:46:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:46:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:46:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:46:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:46:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:46:08.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:46:08.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:46:08.573 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:08.573 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:46:08.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:46:08.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:08.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:46:08.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T15:46:08.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T15:46:08.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T15:46:08.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:08.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:46:08.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-09T15:46:08.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-09T15:46:08.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-03-09T15:46:08.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:08.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:46:09.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-09T15:46:09.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-09T15:46:09.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509448' 2024-03-09T15:46:09.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:09.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T15:46:09.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:09.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:46:09.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T15:46:09.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:09.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T15:46:09.085 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836498 2024-03-09T15:46:09.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T15:46:09.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:09.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-09T15:46:09.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:46:10.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:46:10.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:10.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-03-09T15:46:10.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:10.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-09T15:46:10.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:10.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:46:10.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-09T15:46:10.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:10.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-09T15:46:10.662 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672972 2024-03-09T15:46:10.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-09T15:46:10.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:10.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-09T15:46:10.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:10.965 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-09T15:46:10.965 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:10.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:46:10.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-09T15:46:10.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:10.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-09T15:46:10.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-09T15:46:10.968 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509448 2024-03-09T15:46:10.968 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:46:11.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-09T15:46:11.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:46:11.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:11.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:11.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:46:11.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:11.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:11.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:46:11.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:11.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:11.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:12.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:46:12.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T15:46:12.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:46:12.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T15:46:12.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:46:12.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:12.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:12.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:46:12.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:12.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:12.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:46:12.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:46:12.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:46:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:46:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:46:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T15:46:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:46:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T15:46:13.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:46:13.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:13.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:13.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:46:13.716 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:13.716 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:13.716 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:14.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:46:14.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:46:14.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:46:14.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:46:14.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:46:14.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T15:46:14.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:46:14.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T15:46:14.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:46:14.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:14.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:15.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:46:15.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:15.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:15.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-03-09T15:46:15.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-03-09T15:46:15.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:15.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:15.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:15.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:15.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:15.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:15.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:15.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:15.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:15.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:16.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:16.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:16.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:16.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T15:46:16.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T15:46:16.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-09T15:46:16.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:46:16.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:46:16.759 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:46:16.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:46:16.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:46:16.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:16.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:46:17.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:46:17.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:17.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:46:17.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:46:17.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:46:17.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:46:17.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-09T15:46:17.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:46:17.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:46:17.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:46:17.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:46:17.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:46:17.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-09T15:46:18.955 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T15:46:19.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:46:19.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:46:19.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:46:19.475 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:46:19.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:46:19.506 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:19.506+0000 7f2a053bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:19.508 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:19.509+0000 7f2a053bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:19.509 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:19.510+0000 7f2a053bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:46:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:46:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:46:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:20.024 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:20.024+0000 7f2a053bb640 -1 Falling back to public interface 2024-03-09T15:46:20.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:21.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:22.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:22.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:22.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:46:22.336 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:46:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:22.404 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:22.405+0000 7f2a053bb640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:46:22.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:23.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:23.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:23.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:46:23.627 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:46:23.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:23.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4152402173,v1:127.0.0.1:6803/4152402173] [v2:127.0.0.1:6804/4152402173,v1:127.0.0.1:6805/4152402173] exists,up 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:46:23.972 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:46:23.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:46:23.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:46:23.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:46:24.056 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:46:24.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:46:24.323 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:24.323 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:46:24.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:46:24.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:24.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:46:24.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T15:46:24.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T15:46:24.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T15:46:24.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:24.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:46:24.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-09T15:46:24.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-09T15:46:24.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-03-09T15:46:24.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:24.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509452' 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T15:46:24.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:24.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:46:24.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T15:46:24.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:24.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T15:46:24.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T15:46:24.638 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182402 2024-03-09T15:46:24.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:24.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T15:46:24.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:46:25.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:46:25.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:26.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T15:46:26.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:26.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-09T15:46:26.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:46:26.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-09T15:46:26.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:26.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-09T15:46:26.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-09T15:46:26.201 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672976 2024-03-09T15:46:26.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:26.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-09T15:46:26.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:26.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-09T15:46:26.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:26.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:46:26.473 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-09T15:46:26.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:26.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-09T15:46:26.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-09T15:46:26.475 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509452 2024-03-09T15:46:26.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:46:26.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-09T15:46:26.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:46:26.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:26.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:27.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:27.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:27.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:46:27.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:27.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:27.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:27.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:46:28.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:46:28.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:46:28.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:46:28.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:46:28.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-03-09T15:46:30.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:30.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:46:30.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:46:30.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:30.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:46:30.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:46:30.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:46:30.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:46:30.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:46:30.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:30.082+0000 7fc19040c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:30.084 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:30.085+0000 7fc19040c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:30.085 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:30.086+0000 7fc19040c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:30.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:30.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:30.613 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:30.614+0000 7fc19040c640 -1 Falling back to public interface 2024-03-09T15:46:31.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:31.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:31.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:46:31.606 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:31.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:31.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:32.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:32.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:32.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:46:32.876 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:46:32.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:32.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:33.014 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:33.014+0000 7fc19040c640 -1 osd.1 26 log_to_monitors true 2024-03-09T15:46:33.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:34.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1943684391,v1:127.0.0.1:6811/1943684391] [v2:127.0.0.1:6812/1943684391,v1:127.0.0.1:6813/1943684391] exists,up 89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:46:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:46:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:46:34.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:46:34.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:34.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:46:35.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-09T15:46:35.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-09T15:46:35.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-09T15:46:35.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:35.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:46:35.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T15:46:35.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T15:46:35.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-03-09T15:46:35.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:35.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509455' 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-09T15:46:35.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:35.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:46:35.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-09T15:46:35.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:35.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-09T15:46:35.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-09T15:46:35.239 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182405 2024-03-09T15:46:35.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:35.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-03-09T15:46:35.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:46:36.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:46:36.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:36.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-09T15:46:36.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:36.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-09T15:46:36.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:36.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:46:36.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-09T15:46:36.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:36.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T15:46:36.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-09T15:46:36.792 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 128849018882 2024-03-09T15:46:36.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:37.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T15:46:37.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:37.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T15:46:37.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:37.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:46:37.066 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T15:46:37.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:37.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T15:46:37.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T15:46:37.068 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509455 2024-03-09T15:46:37.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:46:37.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T15:46:37.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:46:37.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:37.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:37.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:37.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:37.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:46:37.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:37.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:37.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:38.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:38.284 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:46:38.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:46:38.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:38.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:46:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:46:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:46:39.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:46:39.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T15:46:40.324 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:46:40.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:46:40.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:46:40.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:46:40.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:46:40.876 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:40.875+0000 7f3287662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:40.900 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:40.900+0000 7f3287662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:40.901+0000 7f3287662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:46:41.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:41.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:41.415 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:41.415+0000 7f3287662640 -1 Falling back to public interface 2024-03-09T15:46:41.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:42.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:42.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:43.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:43.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:43.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:46:43.725 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:46:43.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:43.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:43.755 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:43.754+0000 7f3287662640 -1 osd.0 31 log_to_monitors true 2024-03-09T15:46:44.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:45.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/1454452114,v1:127.0.0.1:6803/1454452114] [v2:127.0.0.1:6804/1454452114,v1:127.0.0.1:6805/1454452114] exists,up 125d7f8f-8901-4f3f-8bf9-8cfbbbd863d6 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:46:45.356 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:46:45.357 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:46:45.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:46:45.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:46:45.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:46:45.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:45.739 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:46:45.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T15:46:45.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T15:46:45.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-09T15:46:45.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:45.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:46:45.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-09T15:46:45.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-09T15:46:45.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-03-09T15:46:45.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:45.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:46:46.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T15:46:46.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T15:46:46.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509458' 2024-03-09T15:46:46.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:46.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-09T15:46:46.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:46.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:46:46.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-09T15:46:46.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:46.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T15:46:46.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-09T15:46:46.053 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 150323855362 2024-03-09T15:46:46.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:46.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T15:46:46.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:46.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-03-09T15:46:46.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:46.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:46:46.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-03-09T15:46:46.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:46.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-09T15:46:46.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-03-09T15:46:46.325 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 128849018885 2024-03-09T15:46:46.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:46.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-09T15:46:46.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:46:47.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:46:47.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:47.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-09T15:46:47.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:46:48.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:46:48.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:49.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-09T15:46:49.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:49.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T15:46:49.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:49.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:46:49.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T15:46:49.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:49.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T15:46:49.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T15:46:49.165 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509458 2024-03-09T15:46:49.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:46:49.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509458 2024-03-09T15:46:49.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:46:49.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:49.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:49.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:50.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:46:50.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:50.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:50.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:50.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:46:50.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:46:50.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:46:50.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-09T15:46:50.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:46:50.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:46:50.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:46:50.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:46:50.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:46:50.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:46:50.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:46:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T15:46:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:46:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:46:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:46:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:46:50.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:46:50.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-09T15:46:52.166 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-03-09T15:46:52.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:52.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:46:52.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:46:52.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:46:52.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:46:52.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:46:52.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:46:52.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:46:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:46:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:46:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:46:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:46:52.715 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:52.715+0000 7fbe2f0c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:52.717 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:52.717+0000 7fbe2f0c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:52.718 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:52.718+0000 7fbe2f0c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:52.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:46:52.981 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:46:52.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:52.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:53.241 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:53.241+0000 7fbe2f0c8640 -1 Falling back to public interface 2024-03-09T15:46:53.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:54.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:54.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:54.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:46:54.258 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:54.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:54.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:54.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:55.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:46:55.546+0000 7fbe2f0c8640 -1 osd.1 36 log_to_monitors true 2024-03-09T15:46:55.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:46:56.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:46:56.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:46:56.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:46:56.902 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:46:56.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:46:56.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:46:57.182 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/93768232,v1:127.0.0.1:6811/93768232] [v2:127.0.0.1:6812/93768232,v1:127.0.0.1:6813/93768232] exists,up 89d0e760-5d49-49fa-b097-d90b135fe4f1 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:46:57.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:46:57.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:46:57.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:46:57.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:46:57.538 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:46:57.538 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:46:57.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:46:57.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:57.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:46:57.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-09T15:46:57.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-09T15:46:57.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-03-09T15:46:57.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:57.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:46:57.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T15:46:57.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T15:46:57.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-03-09T15:46:57.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:46:57.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:46:57.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-09T15:46:57.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-09T15:46:57.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509462' 2024-03-09T15:46:57.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:57.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-03-09T15:46:57.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:57.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:46:57.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-03-09T15:46:57.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:57.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-09T15:46:57.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-03-09T15:46:57.852 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 150323855365 2024-03-09T15:46:57.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:46:58.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-09T15:46:58.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:58.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-09T15:46:58.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:58.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:46:58.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-09T15:46:58.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:58.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T15:46:58.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-09T15:46:58.128 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 171798691842 2024-03-09T15:46:58.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:46:58.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T15:46:58.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:46:58.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-09T15:46:58.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:46:58.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:46:58.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-09T15:46:58.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:46:58.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-09T15:46:58.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-09T15:46:58.416 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509462 2024-03-09T15:46:58.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:46:58.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-09T15:46:58.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:46:58.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:58.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:46:59.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:46:59.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:46:59.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:46:59.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:46:59.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:46:59.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:46:59.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:46:59.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:46:59.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-09T15:46:59.744 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: { 2024-03-09T15:46:59.744 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T15:46:59.744 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:46:59.759 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: { 2024-03-09T15:46:59.759 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T15:46:59.759 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:46:59.772 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2: { 2024-03-09T15:46:59.772 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T15:46:59.772 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:46:59.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-03-09T15:46:59.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T15:46:59.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T15:46:59.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T15:46:59.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:47:00.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-03-09T15:47:00.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-03-09T15:47:00.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:47:00.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T15:47:00.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T15:47:00.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:47:00.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-03-09T15:47:00.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-03-09T15:47:00.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:00.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:00.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:00.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:00.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:00.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-03-09T15:47:00.754 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:47:00.754 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": true, 2024-03-09T15:47:00.754 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T15:47:00.754 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-02-24T15:45:20.755319+0000" 2024-03-09T15:47:00.754 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:47:00.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:47:00.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:47:00.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:00.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:00.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:00.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:01.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:46:06.555914+0000 '>' 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:01.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:47:02.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:47:02.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:47:02.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:47:02.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:02.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:02.036 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:02.036 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:02.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-24T15:45:20.755319+0000 '>' 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:02.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:47:03.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:47:03.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:47:03.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:47:03.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:03.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:03.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:03.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:03.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-24T15:45:20.755319+0000 '>' 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:03.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:47:04.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:47:04.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:47:04.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:47:04.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:04.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:04.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:04.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:04.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-24T15:45:20.755319+0000 '>' 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:04.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:47:05.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:47:05.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:47:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:47:01.454378+0000 '>' 2024-03-09T15:46:06.555914+0000 2024-03-09T15:47:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:47:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-09T15:47:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:47:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:47:06.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:47:06.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:47:06.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:47:06.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:47:06.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:47:06.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:47:06.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:47:06.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:47:06.542 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:47:06.542 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:47:06.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:47:06.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:06.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:47:06.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-03-09T15:47:06.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-03-09T15:47:06.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368' 2024-03-09T15:47:06.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:06.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:47:06.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T15:47:06.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T15:47:06.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-03-09T15:47:06.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:06.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509465' 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855368 2024-03-09T15:47:06.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:06.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:47:06.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855368 2024-03-09T15:47:06.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:06.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-03-09T15:47:06.855 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 150323855368 2024-03-09T15:47:06.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-03-09T15:47:06.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:47:07.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855368 2024-03-09T15:47:07.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:47:08.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:47:08.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:47:08.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-03-09T15:47:08.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:08.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-09T15:47:08.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:08.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:47:08.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-09T15:47:08.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:08.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T15:47:08.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-09T15:47:08.455 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 171798691845 2024-03-09T15:47:08.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:47:08.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-09T15:47:08.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:08.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-09T15:47:08.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:08.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:47:08.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-09T15:47:08.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:08.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-09T15:47:08.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-09T15:47:08.732 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509465 2024-03-09T15:47:08.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:47:09.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-09T15:47:09.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:47:09.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:47:09.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:47:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:47:09.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:47:09.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:47:09.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:47:09.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:47:09.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:47:09.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:47:09.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:47:09.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:47:09.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-03-09T15:47:09.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:47:09.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:47:10.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:47:10.269 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:47:10.269 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:47:10.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:47:10.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:10.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:47:10.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-03-09T15:47:10.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-03-09T15:47:10.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370' 2024-03-09T15:47:10.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:10.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:47:10.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-03-09T15:47:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-03-09T15:47:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691846' 2024-03-09T15:47:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:47:10.477 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:47:10.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-03-09T15:47:10.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-03-09T15:47:10.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691846 2-64424509466' 2024-03-09T15:47:10.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:10.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855370 2024-03-09T15:47:10.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:10.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:47:10.581 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855370 2024-03-09T15:47:10.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:10.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-03-09T15:47:10.583 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 150323855370 2024-03-09T15:47:10.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-03-09T15:47:10.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:47:10.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855370 2024-03-09T15:47:10.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:47:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:47:11.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:47:12.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-03-09T15:47:12.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:12.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-03-09T15:47:12.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:12.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:47:12.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-03-09T15:47:12.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:12.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-03-09T15:47:12.139 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 171798691846 2024-03-09T15:47:12.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-03-09T15:47:12.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:47:12.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691846 2024-03-09T15:47:12.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:47:12.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-03-09T15:47:12.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:47:12.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:47:12.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-03-09T15:47:12.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:47:12.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-03-09T15:47:12.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-03-09T15:47:12.415 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509466 2024-03-09T15:47:12.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:47:12.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509466 2024-03-09T15:47:12.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-03-09T15:47:12.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-09T15:47:12.945 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:47:12.945 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-09T15:47:01.454470+0000 21'10 41:97 [1,0] 1 [1,0] 1 21'10 2024-03-09T15:47:01.454378+0000 21'10 2024-03-09T15:47:01.454378+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:47:01.454378+0000 10 0 2024-03-09T15:47:12.945 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:47:12.945 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:47:12.946 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:47:12.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-09T15:47:12.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-03-09T15:47:13.212 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:47:13.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:47:13.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:47:13.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:47:13.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:47:13.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:47:13.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:47:13.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:47:13.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:47:13.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:47:13.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:47:13.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:47:13.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:47:13.332 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:47:13.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:47:13.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:47:13.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:47:13.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:47:13.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:47:13.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:47:13.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:47:13.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:47:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:47:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:47:13.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:47:13.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:47:13.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:47:13.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:47:13.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:47:13.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:47:13.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:47:13.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:47:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:47:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:47:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:47:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:47:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:47:13.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:47:13.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:47:13.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:47:13.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:47:13.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:47:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:47:13.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:47:13.703 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:47:13.703 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:47:13.703 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:47:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:47:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:47:13.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:47:13.905 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:47:13.906 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:47:13.906 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:47:13.906 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.907 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.907 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:47:13.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:47:13.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:47:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:47:13.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:47:14.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:47:14.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:14.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:14.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:47:14.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:47:14.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:47:14.900 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-03-09T15:47:14.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T15:47:14.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:47:14.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:47:14.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:47:14.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:47:14.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7537855e-9c5c-48aa-a6cf-61ac0a27aed7' 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:47:14.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:47:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCChOxlWMb+NhAAF9sDUGIzWJ5lLR0spMFY9Q== 2024-03-09T15:47:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCChOxlWMb+NhAAF9sDUGIzWJ5lLR0spMFY9Q=="}' 2024-03-09T15:47:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 -i td/osd-scrub-repair/0/new.json 2024-03-09T15:47:15.145 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:47:15.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:47:15.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCChOxlWMb+NhAAF9sDUGIzWJ5lLR0spMFY9Q== --osd-uuid 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:47:15.186 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.187+0000 7f286292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:15.195 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.196+0000 7f286292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:15.197 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.197+0000 7f286292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:15.197 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.198+0000 7f286292a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:47:15.197 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.198+0000 7f286292a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:47:15.197 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.198+0000 7f286292a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:47:15.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:15.288+0000 7f286292a640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:47:20.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:47:20.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:47:20.869 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:47:20.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:47:20.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:47:21.227 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:47:21.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:47:21.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:47:21.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:47:21.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:47:21.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:47:21.256 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:21.256+0000 7f573ca1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:21.260 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:21.260+0000 7f573ca1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:21.261 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:21.261+0000 7f573ca1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:21.520 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:47:21.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:47:21.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:21.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:21.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:21.793 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:21.794+0000 7f573ca1c640 -1 Falling back to public interface 2024-03-09T15:47:22.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:22.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:22.794 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:47:22.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:47:22.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:22.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:23.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:24.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:24.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:24.077 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:47:24.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:47:24.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:24.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:24.118 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:24.119+0000 7f573ca1c640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:47:24.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:25.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:25.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:25.394 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:47:25.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:47:25.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:25.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:25.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:26.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:26.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:26.698 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:47:26.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:47:26.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:26.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:27.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:28.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:28.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:28.073 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:47:28.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:47:28.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:28.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:28.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:29.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:29.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:29.355 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:47:29.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:47:29.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:29.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:29.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:30.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:30.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:30.730 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:47:30.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:47:30.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:30.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:31.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:32.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:32.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:32.136 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:47:32.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:47:32.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:32.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:32.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:33.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:33.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:33.420 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:47:33.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:47:33.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:33.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:33.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:34.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:34.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:34.703 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:47:34.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:47:34.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:34.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:35.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:36.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:36.085 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:47:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:47:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:36.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:36.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:37.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:37.722 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:47:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:47:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:38.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:39.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:39.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:39.385 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:47:39.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:47:39.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:39.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:39.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:40.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:40.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:40.975 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:47:40.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:47:40.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:40.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:41.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:42.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:42.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:42.769 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:47:42.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:47:42.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:42.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:43.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:44.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:44.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:44.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:47:44.238 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:47:44.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:44.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:44.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:45.890 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:47:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:47:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:47:46.228 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/210453688,v1:127.0.0.1:6803/210453688] [v2:127.0.0.1:6804/210453688,v1:127.0.0.1:6805/210453688] exists,up 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:47:46.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:47:46.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:47:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82f05ab3-5b3f-4c4a-b44b-455c137080f2 2024-03-09T15:47:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 82f05ab3-5b3f-4c4a-b44b-455c137080f2' 2024-03-09T15:47:46.236 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 82f05ab3-5b3f-4c4a-b44b-455c137080f2 2024-03-09T15:47:46.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:47:46.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCihOxl34DrDhAA494vkf3p5l0l2Ly/W1dtPw== 2024-03-09T15:47:46.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCihOxl34DrDhAA494vkf3p5l0l2Ly/W1dtPw=="}' 2024-03-09T15:47:46.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82f05ab3-5b3f-4c4a-b44b-455c137080f2 -i td/osd-scrub-repair/1/new.json 2024-03-09T15:47:46.581 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:47:46.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:47:46.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCihOxl34DrDhAA494vkf3p5l0l2Ly/W1dtPw== --osd-uuid 82f05ab3-5b3f-4c4a-b44b-455c137080f2 2024-03-09T15:47:46.624 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.624+0000 7f0710ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:46.665 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.665+0000 7f0710ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:46.666 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.666+0000 7f0710ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:46.666 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.666+0000 7f0710ab5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:47:46.667 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.666+0000 7f0710ab5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:47:46.667 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.666+0000 7f0710ab5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:47:46.740 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:46.740+0000 7f0710ab5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:47:52.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:47:52.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:47:52.922 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:47:52.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:47:52.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:47:53.302 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:47:53.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:47:53.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:47:53.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:47:53.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:47:53.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:47:53.331 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:53.331+0000 7f490431f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:53.333+0000 7f490431f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:53.334+0000 7f490431f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:47:53.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:53.577 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:47:53.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:47:53.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:53.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:47:53.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:53.873 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:53.873+0000 7f490431f640 -1 Falling back to public interface 2024-03-09T15:47:54.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:54.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:54.850 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:47:54.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:47:54.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:54.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:47:55.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:56.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:56.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:56.126 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:47:56.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:47:56.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:56.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:47:56.329 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:47:56.328+0000 7f490431f640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:47:56.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:57.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:57.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:57.421 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:47:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:47:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:47:57.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:58.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:58.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:58.705 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:47:58.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:47:58.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:58.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:47:58.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:47:59.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:47:59.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:47:59.994 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:47:59.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:47:59.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:47:59.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:00.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:01.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:01.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:01.311 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:48:01.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:48:01.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:01.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:01.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:02.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:02.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:02.593 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:48:02.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:48:02.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:02.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:02.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:03.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:03.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:03.928 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:48:03.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:48:03.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:03.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:05.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:05.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:05.300 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:48:05.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:48:05.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:05.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:05.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:06.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:06.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:06.582 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:48:06.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:48:06.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:06.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:06.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:07.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:07.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:07.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:48:07.856 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:48:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:08.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:09.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:09.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:09.236 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:48:09.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:48:09.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:09.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:10.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:48:11.100 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:48:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:12.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:13.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:13.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:13.229 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:48:13.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:48:13.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:13.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:13.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:14.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:14.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:14.758 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:48:14.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:48:14.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:14.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:15.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:16.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:16.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:16.439 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:48:16.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:48:16.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:16.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:17.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:18.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:18.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:18.153 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:48:18.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:48:18.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:18.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:18.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:19.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:19.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:19.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:48:19.697 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:48:19.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:19.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:48:19.989 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1666020925,v1:127.0.0.1:6811/1666020925] [v2:127.0.0.1:6812/1666020925,v1:127.0.0.1:6813/1666020925] exists,up 82f05ab3-5b3f-4c4a-b44b-455c137080f2 2024-03-09T15:48:19.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:48:19.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:48:19.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:48:19.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T15:48:19.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:48:19.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:48:19.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:48:19.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:48:19.998 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 65afc5f5-154e-4067-af20-a212a9dbd641 2024-03-09T15:48:19.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65afc5f5-154e-4067-af20-a212a9dbd641 2024-03-09T15:48:19.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 65afc5f5-154e-4067-af20-a212a9dbd641' 2024-03-09T15:48:19.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:48:20.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEhOxlk16uABAAg5FDIkyVhgnr6FFdsQZfyw== 2024-03-09T15:48:20.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEhOxlk16uABAAg5FDIkyVhgnr6FFdsQZfyw=="}' 2024-03-09T15:48:20.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65afc5f5-154e-4067-af20-a212a9dbd641 -i td/osd-scrub-repair/2/new.json 2024-03-09T15:48:20.355 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:48:20.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:48:20.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDEhOxlk16uABAAg5FDIkyVhgnr6FFdsQZfyw== --osd-uuid 65afc5f5-154e-4067-af20-a212a9dbd641 2024-03-09T15:48:20.397 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.398+0000 7f8a87c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:20.440 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.440+0000 7f8a87c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:20.440 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.441+0000 7f8a87c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:20.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.442+0000 7f8a87c0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:48:20.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.442+0000 7f8a87c0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:48:20.441 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.442+0000 7f8a87c0d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:48:20.506 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:20.507+0000 7f8a87c0d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:48:26.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:48:26.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:48:26.389 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:48:26.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:48:26.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:48:26.742 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:48:26.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:48:26.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:48:26.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:48:26.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:48:26.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:48:26.771 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:26.771+0000 7f317ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:26.773 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:26.773+0000 7f317ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:26.774 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:26.774+0000 7f317ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:48:27.015 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:48:27.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:27.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:27.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:27.304 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:27.305+0000 7f317ca15640 -1 Falling back to public interface 2024-03-09T15:48:28.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:28.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:28.286 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:48:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:48:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:28.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:29.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:29.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:29.569 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:48:29.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:48:29.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:29.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:29.587 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:48:29.588+0000 7f317ca15640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:48:29.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:30.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:30.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:30.914 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:48:30.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:48:30.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:30.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:31.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:32.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:32.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:32.232 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:48:32.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:48:32.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:32.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:32.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:33.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:33.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:33.512 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:48:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:48:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:33.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:33.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:34.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:34.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:34.797 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:48:34.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:48:34.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:34.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:35.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:36.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:36.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:36.109 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:48:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:48:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:36.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:37.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:37.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:37.417 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:48:37.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:48:37.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:37.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:37.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:38.701 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:48:38.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:48:38.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:38.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:38.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:39.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:39.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:39.984 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:48:39.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:48:39.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:39.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:41.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:41.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:41.268 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:48:41.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:48:41.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:41.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:41.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:42.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:42.566 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:48:42.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:42.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:48:42.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:42.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:43.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:44.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:44.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:44.134 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:48:44.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:48:44.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:44.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:44.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:45.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:45.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:45.703 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:48:45.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:48:45.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:45.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:46.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:47.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:47.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:47.382 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:48:47.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:48:47.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:47.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:47.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:48.674 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:48:48.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:48:48.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:48.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:48.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:49.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:49.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:49.955 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:48:49.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:48:49.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:49.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:50.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:48:51.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:48:51.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:48:51.413 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:48:51.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:48:51.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:48:51.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:48:51.739 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2446107655,v1:127.0.0.1:6819/2446107655] [v2:127.0.0.1:6820/2446107655,v1:127.0.0.1:6821/2446107655] exists,up 65afc5f5-154e-4067-af20-a212a9dbd641 2024-03-09T15:48:51.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:48:51.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:48:51.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:48:51.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-03-09T15:48:51.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T15:48:52.163 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T15:48:52.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:48:53.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-03-09T15:48:53.807 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:48:53.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:48:53.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:48:53.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:48:53.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:48:53.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:48:53.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:48:53.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:48:53.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:48:53.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:48:54.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:48:54.264 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:48:54.264 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:48:54.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:48:54.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:48:54.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:48:54.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-09T15:48:54.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-09T15:48:54.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-09T15:48:54.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:48:54.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:48:54.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-09T15:48:54.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-09T15:48:54.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672969' 2024-03-09T15:48:54.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:48:54.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509443 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509443 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672969 2-64424509443' 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-09T15:48:54.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:48:54.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:48:54.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-09T15:48:54.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:48:54.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-09T15:48:54.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-09T15:48:54.638 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836496 2024-03-09T15:48:54.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:48:54.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-09T15:48:54.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:48:55.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:48:55.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:48:56.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-03-09T15:48:56.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:48:56.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-09T15:48:56.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:48:56.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:48:56.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-09T15:48:56.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:48:56.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-09T15:48:56.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-09T15:48:56.209 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672969 2024-03-09T15:48:56.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:48:56.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-03-09T15:48:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:48:56.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509443 2024-03-09T15:48:56.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:48:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:48:56.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509443 2024-03-09T15:48:56.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:48:56.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509443 2024-03-09T15:48:56.485 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509443 2024-03-09T15:48:56.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-03-09T15:48:56.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:48:56.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509443 2024-03-09T15:48:56.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:48:56.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:48:56.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:48:57.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:48:57.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:48:57.388 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:48:57.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:48:57.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:48:57.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:48:57.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:48:57.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:48:57.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-03-09T15:48:57.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-03-09T15:48:57.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:48:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:48:58.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:48:58.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:48:58.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:48:58.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:48:58.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:48:58.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:48:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T15:48:59.849 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T15:49:00.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:49:00.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:49:00.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:49:00.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:49:00.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:49:00.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:49:00.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:49:00.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:49:00.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:00.398+0000 7fcf69b38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:00.401 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:00.401+0000 7fcf69b38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:00.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:00.402+0000 7fcf69b38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:00.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:00.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:00.925 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:00.925+0000 7fcf69b38640 -1 Falling back to public interface 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:02.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:03.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:03.338 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:03.337+0000 7fcf69b38640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:49:03.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:04.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:04.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:04.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:49:04.545 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:49:04.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:04.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:04.827 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2291405788,v1:127.0.0.1:6803/2291405788] [v2:127.0.0.1:6804/2291405788,v1:127.0.0.1:6805/2291405788] exists,up 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:49:04.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:49:04.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:49:04.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:49:04.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:49:05.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:49:05.200 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:49:05.200 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:49:05.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:49:05.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:05.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:49:05.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T15:49:05.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T15:49:05.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T15:49:05.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:05.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:49:05.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T15:49:05.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T15:49:05.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672973' 2024-03-09T15:49:05.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:05.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:49:05.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T15:49:05.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T15:49:05.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672973 2-64424509446' 2024-03-09T15:49:05.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:05.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T15:49:05.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:05.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:49:05.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T15:49:05.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:05.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T15:49:05.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T15:49:05.517 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182402 2024-03-09T15:49:05.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:05.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T15:49:05.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:49:06.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:49:06.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:07.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T15:49:07.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:49:08.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:49:08.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:08.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T15:49:08.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:08.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T15:49:08.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:08.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:49:08.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T15:49:08.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:08.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T15:49:08.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T15:49:08.354 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672973 2024-03-09T15:49:08.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:49:08.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T15:49:08.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:08.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T15:49:08.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:08.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:49:08.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T15:49:08.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:08.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T15:49:08.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T15:49:08.626 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509446 2024-03-09T15:49:08.627 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:49:08.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-09T15:49:08.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:49:08.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:08.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:49:09.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:49:09.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:49:09.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:49:09.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:09.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:49:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:49:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:49:09.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-03-09T15:49:09.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T15:49:09.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:49:09.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T15:49:09.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:49:10.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-03-09T15:49:10.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-03-09T15:49:10.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:49:10.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:49:10.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:49:10.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:49:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-03-09T15:49:10.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-03-09T15:49:10.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:10.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:10.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:10.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-03-09T15:49:10.804 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T15:49:10.804 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": false, 2024-03-09T15:49:10.804 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T15:49:10.804 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-03-02T15:47:30.805653+0000" 2024-03-09T15:49:10.804 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:10.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:11.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:48:52.047537+0000 '>' 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:11.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:49:12.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:49:12.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:49:12.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:49:12.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:12.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:12.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:12.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:12.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:48:52.047537+0000 '>' 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:12.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:49:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:49:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:49:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:49:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:13.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:13.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:48:52.047537+0000 '>' 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:13.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:49:14.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:49:14.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:49:14.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:49:14.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:14.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:14.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:14.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:14.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:48:52.047537+0000 '>' 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:14.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:49:15.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:49:15.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:49:15.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T15:49:15.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:49:15.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:49:15.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:49:15.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:49:16.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:49:12.664267+0000 '>' 2024-03-09T15:48:52.047537+0000 2024-03-09T15:49:16.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:49:16.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T15:49:16.495 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:49:16.496 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:49:12.822333+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:49:12.664267+0000 21'1 2024-03-09T15:49:12.664267+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:49:12.664267+0000 1 0 2024-03-09T15:49:16.496 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:49:16.496 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:49:16.496 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:49:16.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-09T15:49:18.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T15:49:18.774 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:49:18.774 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:49:12.822333+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:49:12.664267+0000 21'1 2024-03-09T15:49:12.664267+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:49:12.664267+0000 1 0 2024-03-09T15:49:18.774 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:49:18.774 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:49:18.774 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:49:18.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-09T15:49:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T15:49:21.053 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:49:21.053 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:49:12.822333+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:49:12.664267+0000 21'1 2024-03-09T15:49:12.664267+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:49:12.664267+0000 1 0 2024-03-09T15:49:21.053 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:49:21.053 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:49:21.053 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:49:21.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-03-09T15:49:26.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-09T15:49:26.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:49:26.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:49:26.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:49:26.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:49:26.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:49:26.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:49:26.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:49:26.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:49:26.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:49:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:49:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:49:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:49:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:26.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:49:26.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-09T15:49:26.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-09T15:49:26.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-09T15:49:26.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:26.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:49:26.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T15:49:26.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T15:49:26.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672978' 2024-03-09T15:49:26.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:26.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:49:26.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-09T15:49:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-09T15:49:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672978 2-64424509451' 2024-03-09T15:49:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:26.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-09T15:49:26.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:26.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:49:26.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-09T15:49:26.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:26.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-09T15:49:26.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-09T15:49:26.748 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 107374182407 2024-03-09T15:49:26.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:27.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-03-09T15:49:27.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:49:28.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:49:28.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:28.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-03-09T15:49:28.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:28.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T15:49:28.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:28.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:49:28.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T15:49:28.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:28.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T15:49:28.304 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672978 2024-03-09T15:49:28.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T15:49:28.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:49:28.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-09T15:49:28.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:28.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-09T15:49:28.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:28.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:49:28.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-09T15:49:28.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:28.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-09T15:49:28.578 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509451 2024-03-09T15:49:28.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-09T15:49:28.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:49:28.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-09T15:49:28.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:49:28.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:28.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:29.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:49:29.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:49:29.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:49:29.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:49:29.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:49:29.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:49:29.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:49:29.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:49:29.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:49:29.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:49:29.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:29.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:29.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:49:29.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:49:29.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:49:29.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T15:49:30.065 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:49:30.065 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:49:12.822333+0000 21'1 26:68 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:49:12.664267+0000 21'1 2024-03-09T15:49:12.664267+0000 0 1 periodic scrub scheduled @ 2024-03-10T15:49:12.664267+0000 1 0 2024-03-09T15:49:30.065 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:49:30.065 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:49:30.065 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:49:30.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:49:30.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:49:30.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:49:30.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:49:30.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:49:30.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:49:30.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:49:30.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:49:30.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:49:30.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:49:30.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:49:30.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:49:30.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:49:30.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:49:30.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:49:30.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:49:30.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:49:30.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T15:49:32.200 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:49:32.200 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:49:32.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:49:32.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:49:32.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:49:32.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:49:32.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:32.469 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:49:32.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:49:32.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:49:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:49:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:49:32.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:49:32.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:32.500+0000 7ff9896f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:32.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:32.502+0000 7ff9896f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:32.503 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:32.504+0000 7ff9896f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:32.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:32.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:33.021 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:33.022+0000 7ff9896f5640 -1 Falling back to public interface 2024-03-09T15:49:33.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:34.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:34.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:34.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:49:34.026 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:49:34.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:34.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:34.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:35.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:35.378 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:35.379+0000 7ff9896f5640 -1 osd.0 26 log_to_monitors true 2024-03-09T15:49:35.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:49:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:49:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:36.865 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/208449641,v1:127.0.0.1:6803/208449641] [v2:127.0.0.1:6804/208449641,v1:127.0.0.1:6805/208449641] exists,up 7537855e-9c5c-48aa-a6cf-61ac0a27aed7 2024-03-09T15:49:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:49:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:49:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:49:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:49:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:49:36.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:49:36.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:37.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:49:37.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T15:49:37.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T15:49:37.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T15:49:37.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:37.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:49:37.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-09T15:49:37.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-09T15:49:37.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981' 2024-03-09T15:49:37.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:49:37.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981 2-64424509455' 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T15:49:37.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:37.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:49:37.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T15:49:37.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:37.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T15:49:37.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T15:49:37.539 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 128849018882 2024-03-09T15:49:37.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:37.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:49:37.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:49:38.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:49:38.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:49:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T15:49:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:39.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-09T15:49:39.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:39.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:49:39.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-09T15:49:39.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:39.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-09T15:49:39.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-09T15:49:39.099 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672981 2024-03-09T15:49:39.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:49:39.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-09T15:49:39.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:49:39.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T15:49:39.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:49:39.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:49:39.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T15:49:39.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:49:39.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T15:49:39.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T15:49:39.376 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509455 2024-03-09T15:49:39.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:49:39.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T15:49:39.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:49:39.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:39.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:40.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:49:40.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:49:40.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:49:40.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:49:40.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:49:40.009 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:49:40.009 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:49:40.009 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:49:40.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:49:40.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:49:40.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:49:40.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:49:40.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:49:40.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:49:40.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:49:40.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:49:40.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:49:40.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:49:40.644 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:11.632+0000 7f48e8298640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-03-09T15:49:40.644 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:11.632+0000 7f48e8298640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-03-09T15:49:40.644 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:11.632+0000 7f48e8298640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T15:49:40.645 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:12.663+0000 7f48e8298640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:49:40.645 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:12.663+0000 7f48e8298640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:49:40.645 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:12.663+0000 7f48e8298640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T15:49:40.645 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:12.663+0000 7f48e8298640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T15:49:40.645 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:49:12.663+0000 7f48e8298640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:49:40.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-03-09T15:49:40.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:49:40.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:49:40.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:49:40.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:49:40.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:49:40.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:49:40.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:49:40.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:49:40.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:49:40.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:49:40.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:49:40.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:49:40.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:49:40.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:49:40.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:49:40.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:49:40.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:49:40.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:49:40.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:49:40.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:49:40.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:49:40.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:40.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:40.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:49:40.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:49:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:49:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:49:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:49:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:49:40.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:49:40.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:49:40.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:49:40.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:49:40.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:49:40.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:49:40.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:49:40.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:49:40.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:49:40.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:49:40.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:49:40.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:49:40.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:49:40.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:49:40.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:40.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:40.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:49:40.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:49:40.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:49:40.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:49:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:49:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:49:40.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:49:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:49:41.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:41.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:49:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:49:41.407 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:49:41.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:49:41.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:49:41.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:49:41.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:49:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:49:42.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:49:42.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:42.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:42.374 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:42.374 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:42.375 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:42.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:42.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:49:42.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:49:42.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:49:42.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:49:42.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:42.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:49:42.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5276e770-3ac0-4fc8-ab2c-37984e3f646a' 2024-03-09T15:49:42.410 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:49:42.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:49:42.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWhexlWVdWGRAAui2gnWWvvhlrivPpm3uskA== 2024-03-09T15:49:42.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWhexlWVdWGRAAui2gnWWvvhlrivPpm3uskA=="}' 2024-03-09T15:49:42.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5276e770-3ac0-4fc8-ab2c-37984e3f646a -i td/osd-scrub-repair/0/new.json 2024-03-09T15:49:42.629 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:49:42.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:49:42.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAWhexlWVdWGRAAui2gnWWvvhlrivPpm3uskA== --osd-uuid 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:49:42.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.672+0000 7fe9e1a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:42.680 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.680+0000 7fe9e1a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:42.681 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.681+0000 7fe9e1a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:42.681 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.682+0000 7fe9e1a0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:49:42.681 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.682+0000 7fe9e1a0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:49:42.681 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.682+0000 7fe9e1a0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:49:42.772 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:42.773+0000 7fe9e1a0e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:49:48.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:49:48.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:49:48.331 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:49:48.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:49:48.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:49:48.689 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:49:48.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:49:48.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:49:48.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:49:48.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:49:48.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:49:48.718 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:48.719+0000 7f3094c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:48.720 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:48.721+0000 7f3094c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:48.721 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:48.722+0000 7f3094c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:48.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:49:48.974 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:49:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:49.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:49.249 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:49.249+0000 7f3094c0d640 -1 Falling back to public interface 2024-03-09T15:49:50.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:50.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:50.248 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:49:50.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:49:50.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:50.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:50.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:51.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:51.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:51.525 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:49:51.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:49:51.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:51.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:51.568 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:51.569+0000 7f3094c0d640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:49:51.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:49:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:49:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:49:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:49:52.845 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:49:52.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:49:52.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:49:53.131 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2795616758,v1:127.0.0.1:6803/2795616758] [v2:127.0.0.1:6804/2795616758,v1:127.0.0.1:6805/2795616758] exists,up 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:49:53.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:49:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:49:53.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 04500b8c-6d91-42e7-b694-c63036b43ffc 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04500b8c-6d91-42e7-b694-c63036b43ffc 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 04500b8c-6d91-42e7-b694-c63036b43ffc' 2024-03-09T15:49:53.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:49:53.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhhexl9tIbCRAA5f39bsLFEqmcASqN2GriXA== 2024-03-09T15:49:53.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhhexl9tIbCRAA5f39bsLFEqmcASqN2GriXA=="}' 2024-03-09T15:49:53.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04500b8c-6d91-42e7-b694-c63036b43ffc -i td/osd-scrub-repair/1/new.json 2024-03-09T15:49:53.471 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:49:53.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:49:53.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAhhexl9tIbCRAA5f39bsLFEqmcASqN2GriXA== --osd-uuid 04500b8c-6d91-42e7-b694-c63036b43ffc 2024-03-09T15:49:53.513 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.514+0000 7faee8a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:53.564 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.564+0000 7faee8a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:53.564 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.565+0000 7faee8a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:53.565 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.566+0000 7faee8a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:49:53.565 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.566+0000 7faee8a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:49:53.565 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.566+0000 7faee8a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:49:53.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:53.639+0000 7faee8a0e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:49:59.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:49:59.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:49:59.418 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:49:59.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:49:59.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:49:59.776 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:49:59.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:49:59.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:49:59.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:49:59.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:49:59.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:49:59.805 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:59.805+0000 7f21b2a08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:59.807 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:59.807+0000 7f21b2a08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:49:59.808 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:49:59.808+0000 7f21b2a08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:00.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:50:00.088 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:50:00.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:00.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:50:00.328 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:00.328+0000 7f21b2a08640 -1 Falling back to public interface 2024-03-09T15:50:00.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:01.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:01.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:01.362 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:50:01.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:50:01.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:01.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:50:01.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:02.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:02.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:02.638 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:50:02.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:50:02.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:02.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:50:02.795 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:02.795+0000 7f21b2a08640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:50:02.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:03.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:03.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:03.989 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:50:03.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:50:03.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:03.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:50:04.263 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1235404303,v1:127.0.0.1:6811/1235404303] [v2:127.0.0.1:6812/1235404303,v1:127.0.0.1:6813/1235404303] exists,up 04500b8c-6d91-42e7-b694-c63036b43ffc 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:04.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:50:04.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:50:04.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:50:04.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a98003a-1f13-462a-b3d4-d91314eff2b1 2024-03-09T15:50:04.272 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 4a98003a-1f13-462a-b3d4-d91314eff2b1 2024-03-09T15:50:04.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4a98003a-1f13-462a-b3d4-d91314eff2b1' 2024-03-09T15:50:04.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:50:04.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAshexlo13/EBAA4ZTScjnDwH8V4GSieiOPMQ== 2024-03-09T15:50:04.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAshexlo13/EBAA4ZTScjnDwH8V4GSieiOPMQ=="}' 2024-03-09T15:50:04.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a98003a-1f13-462a-b3d4-d91314eff2b1 -i td/osd-scrub-repair/2/new.json 2024-03-09T15:50:04.665 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:50:04.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:50:04.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAshexlo13/EBAA4ZTScjnDwH8V4GSieiOPMQ== --osd-uuid 4a98003a-1f13-462a-b3d4-d91314eff2b1 2024-03-09T15:50:04.708 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.707+0000 7fc0e3808640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:04.741 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.741+0000 7fc0e3808640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:04.742 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.742+0000 7fc0e3808640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:04.742 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.742+0000 7fc0e3808640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:50:04.742 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.742+0000 7fc0e3808640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:50:04.742 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.742+0000 7fc0e3808640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:50:04.808 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:04.808+0000 7fc0e3808640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:50:10.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:50:10.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:50:10.738 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:50:10.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:50:10.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:50:11.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:50:11.089 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:50:11.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:11.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:50:11.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:50:11.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:50:11.118 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:11.117+0000 7ff82be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:11.119 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:11.119+0000 7ff82be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:11.120 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:11.120+0000 7ff82be0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:50:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:50:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:50:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:50:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:50:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:50:11.361 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:50:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:50:11.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:11.641 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:11.641+0000 7ff82be0e640 -1 Falling back to public interface 2024-03-09T15:50:12.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:12.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:12.635 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:50:12.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:50:12.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:12.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:50:12.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:13.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:13.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:13.916 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:50:13.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:50:13.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:13.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:50:13.971 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:13.971+0000 7ff82be0e640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:50:14.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:15.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:15.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:15.194 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:50:15.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:50:15.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:15.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:50:15.512 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2987116048,v1:127.0.0.1:6819/2987116048] [v2:127.0.0.1:6820/2987116048,v1:127.0.0.1:6821/2987116048] exists,up 4a98003a-1f13-462a-b3d4-d91314eff2b1 2024-03-09T15:50:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:50:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:50:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:50:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-03-09T15:50:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T15:50:15.874 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T15:50:15.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:50:16.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-03-09T15:50:17.463 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T15:50:17.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-03-09T15:50:17.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:50:17.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:50:17.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:50:17.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:50:17.476 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:50:17.476 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:50:17.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:50:17.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:50:17.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:50:17.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:17.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:50:18.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-03-09T15:50:18.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-03-09T15:50:18.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-03-09T15:50:18.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:18.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:50:18.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-03-09T15:50:18.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-03-09T15:50:18.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-03-09T15:50:18.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:18.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:50:18.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-03-09T15:50:18.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-03-09T15:50:18.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-03-09T15:50:18.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:18.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-03-09T15:50:18.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:18.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:50:18.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-03-09T15:50:18.296 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:18.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-03-09T15:50:18.297 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836487 2024-03-09T15:50:18.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-03-09T15:50:18.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:18.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-03-09T15:50:18.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:50:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:50:19.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:20.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-03-09T15:50:20.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:20.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-03-09T15:50:20.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:20.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:50:20.039 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-03-09T15:50:20.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:20.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-03-09T15:50:20.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-03-09T15:50:20.041 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672964 2024-03-09T15:50:20.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:50:20.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-03-09T15:50:20.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:20.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-03-09T15:50:20.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:20.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:50:20.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-03-09T15:50:20.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:20.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-03-09T15:50:20.316 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509442 2024-03-09T15:50:20.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-03-09T15:50:20.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:50:20.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-03-09T15:50:20.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:50:20.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:20.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:20.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:50:20.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:50:20.968 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:50:20.968 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:50:20.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:50:20.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:50:20.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:50:20.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:50:21.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:50:21.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:50:21.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:21.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-03-09T15:50:21.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:50:21.720 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:50:21.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:21.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:50:21.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:50:21.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:21.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:50:22.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:50:22.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T15:50:22.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:50:22.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:50:22.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:50:22.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:50:22.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:22.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T15:50:23.834 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:50:24.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:24.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:50:24.355 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:50:24.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:24.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:50:24.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:50:24.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:50:24.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:50:24.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:50:24.385 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:24.386+0000 7fa4a000c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:24.387 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:24.388+0000 7fa4a000c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:24.388 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:24.389+0000 7fa4a000c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:24.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:24.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:24.915 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:24.916+0000 7fa4a000c640 -1 Falling back to public interface 2024-03-09T15:50:24.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:25.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:25.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:25.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:50:25.957 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:25.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:25.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:26.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:27.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:27.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:27.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:50:27.237 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:50:27.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:27.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:27.256 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:27.257+0000 7fa4a000c640 -1 osd.0 21 log_to_monitors true 2024-03-09T15:50:27.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:28.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/969674890,v1:127.0.0.1:6803/969674890] [v2:127.0.0.1:6804/969674890,v1:127.0.0.1:6805/969674890] exists,up 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:50:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:50:28.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:50:28.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:50:29.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:50:29.194 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:29.194 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:50:29.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:50:29.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:29.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:50:29.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T15:50:29.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T15:50:29.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T15:50:29.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:29.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:50:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-09T15:50:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-09T15:50:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-03-09T15:50:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:29.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509445' 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:29.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:50:29.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T15:50:29.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:29.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T15:50:29.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T15:50:29.507 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 107374182402 2024-03-09T15:50:29.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:29.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T15:50:29.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:50:30.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:50:30.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:31.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T15:50:31.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:31.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-09T15:50:31.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:31.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:50:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-09T15:50:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:31.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-09T15:50:31.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-09T15:50:31.065 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672968 2024-03-09T15:50:31.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:50:31.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-09T15:50:31.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:31.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T15:50:31.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:31.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:50:31.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T15:50:31.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:31.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T15:50:31.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T15:50:31.339 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509445 2024-03-09T15:50:31.339 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:50:31.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-09T15:50:31.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:50:31.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:31.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:31.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:50:31.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:50:31.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:50:32.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:50:32.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:50:32.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:32.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T15:50:32.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:50:32.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:32.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:50:32.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-03-09T15:50:32.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-03-09T15:50:32.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:50:32.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:50:32.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:32.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:50:33.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-03-09T15:50:33.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-03-09T15:50:33.097 INFO:tasks.workunit.client.0.smithi146.stdout:Affected PG 1.0 w/ primary 1 2024-03-09T15:50:33.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-03-09T15:50:33.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:50:33.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:50:33.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:50:33.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-03-09T15:50:15.829552+0000 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:50:33.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:50:33.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-03-09T15:50:15.829552+0000 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-03-09T15:50:33.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T15:50:33.634 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-03-09T15:50:33.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T15:50:33.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T15:50:33.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-03-09T15:50:33.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T15:50:33.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-03-09T15:50:33.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:50:33.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:34.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:50:34.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-09T15:50:34.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-09T15:50:34.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-09T15:50:34.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:34.180 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:50:34.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-09T15:50:34.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-09T15:50:34.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970' 2024-03-09T15:50:34.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:34.283 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:50:34.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-09T15:50:34.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-09T15:50:34.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970 2-64424509447' 2024-03-09T15:50:34.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:34.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-09T15:50:34.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:34.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:50:34.388 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-09T15:50:34.388 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:34.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-09T15:50:34.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-09T15:50:34.389 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 107374182404 2024-03-09T15:50:34.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:34.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-03-09T15:50:34.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:34.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-09T15:50:34.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:34.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:50:34.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-09T15:50:34.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:34.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-09T15:50:34.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-09T15:50:34.691 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672970 2024-03-09T15:50:34.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:50:34.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-09T15:50:34.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:34.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-09T15:50:34.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:34.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:50:34.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-09T15:50:34.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:34.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-09T15:50:34.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-09T15:50:34.963 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509447 2024-03-09T15:50:34.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:50:35.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-09T15:50:35.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-03-09T15:50:35.266 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09 15:50:35.267707177+00:00 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T15:50:35.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-03-09T15:50:35.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-03-09T15:50:35.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:35.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:35.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:35.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:35.465 INFO:tasks.workunit.client.0.smithi146.stdout:{"deep":true,"must":false,"stamp":"2024-02-24T15:48:55.350725+0000"}0 ) state now: "active+clean" 2024-03-09T15:50:35.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-03-09T15:50:35.465 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:35.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:35.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:35.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-03-09T15:50:35.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:35.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:35.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:35.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:35.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:35.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:35.875 INFO:tasks.workunit.client.0.smithi146.stdout:1 ) state now: "active+clean" 2024-03-09T15:50:35.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-03-09T15:50:35.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:35.876 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:35.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:35.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-03-09T15:50:35.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:36.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:36.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:36.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:36.288 INFO:tasks.workunit.client.0.smithi146.stdout:2 ) state now: "active+clean" 2024-03-09T15:50:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-03-09T15:50:36.288 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:36.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:36.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:36.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-03-09T15:50:36.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:36.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:36.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:36.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:36.697 INFO:tasks.workunit.client.0.smithi146.stdout:3 ) state now: "active+clean" 2024-03-09T15:50:36.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-03-09T15:50:36.697 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-03-09T15:50:36.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:36.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:36.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:36.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:36.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:37.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:37.105 INFO:tasks.workunit.client.0.smithi146.stdout:4 ) state now: "active+clean" 2024-03-09T15:50:37.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-03-09T15:50:37.105 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:37.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:37.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:37.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-03-09T15:50:37.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-03-09T15:50:37.106 INFO:tasks.workunit.client.0.smithi146.stdout:loop --------> 4 2024-03-09T15:50:37.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:37.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:37.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:37.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:37.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:37.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:37.516 INFO:tasks.workunit.client.0.smithi146.stdout:5 ) state now: "active+clean" 2024-03-09T15:50:37.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-03-09T15:50:37.517 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:37.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:37.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:37.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-03-09T15:50:37.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:37.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:37.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:37.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:37.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:37.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T15:50:37.925 INFO:tasks.workunit.client.0.smithi146.stdout:6 ) state now: "active+clean" 2024-03-09T15:50:37.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-03-09T15:50:37.925 INFO:tasks.workunit.client.0.smithi146.stdout:"active+clean" 2024-03-09T15:50:37.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:37.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T15:50:37.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-03-09T15:50:37.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T15:50:38.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T15:50:38.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T15:50:38.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T15:50:38.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T15:50:38.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.334 INFO:tasks.workunit.client.0.smithi146.stdout:7 ) state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:38.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T15:50:38.335 INFO:tasks.workunit.client.0.smithi146.stdout:found scrub 2024-03-09T15:50:38.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-03-09T15:50:38.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-03-09T15:50:38.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-03-09T15:50:38.336 INFO:tasks.workunit.client.0.smithi146.stdout:initiate_and_fetch_state ret: 0 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-03-09T15:50:38.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.td/osd-scrub-repair.asok 2024-03-09T15:50:38.339 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-03-09T15:50:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-03-09T15:50:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-03-09T15:50:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:38.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:38.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.511 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:38.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:38.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-03-09T15:50:38.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T15:50:38.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T15:50:38.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:38.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:38.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:38.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.919 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:38.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:38.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:38.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-03-09T15:50:38.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T15:50:39.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T15:50:39.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:39.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:39.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:39.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:39.327 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:39.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:39.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:39.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-03-09T15:50:39.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T15:50:39.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T15:50:39.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:39.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:39.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:39.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:39.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:39.736 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:39.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:39.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-03-09T15:50:39.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T15:50:40.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T15:50:40.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:40.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:40.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T15:50:40.144 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T15:50:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T15:50:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-03-09T15:50:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-03-09T15:50:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:50:40.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:40.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:50:40.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-03-09T15:50:40.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-03-09T15:50:40.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-03-09T15:50:40.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:40.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:50:40.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-09T15:50:40.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-09T15:50:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972' 2024-03-09T15:50:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:40.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972 2-64424509450' 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-03-09T15:50:40.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:40.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:50:40.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-03-09T15:50:40.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:40.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-03-09T15:50:40.732 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 107374182406 2024-03-09T15:50:40.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-03-09T15:50:40.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:41.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-03-09T15:50:41.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:50:42.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:50:42.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:42.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-03-09T15:50:42.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:50:43.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:50:43.291 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:43.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-03-09T15:50:43.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:43.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-09T15:50:43.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:43.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:50:43.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-09T15:50:43.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:43.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-09T15:50:43.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-09T15:50:43.577 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672972 2024-03-09T15:50:43.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:50:43.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-09T15:50:43.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:43.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T15:50:43.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:43.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:50:43.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T15:50:43.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:43.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T15:50:43.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T15:50:43.851 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509450 2024-03-09T15:50:43.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:50:44.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T15:50:44.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T15:50:44.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T15:50:44.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T15:50:44.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T15:50:44.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T15:50:44.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-03-09T15:50:44.530 INFO:tasks.workunit.client.0.smithi146.stdout:wait-scrub-end state now: "active+clean" 2024-03-09T15:50:44.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-03-09T15:50:44.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:44.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-03-09T15:50:44.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-09T15:50:44.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-03-09T15:50:44.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-03-09T15:50:44.790 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T15:50:44.790 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T15:50:41.277781+0000 21'1 26:56 [1,0] 1 [1,0] 1 21'1 2024-03-09T15:50:40.958447+0000 21'1 2024-03-09T15:50:40.958447+0000 0 4 periodic scrub scheduled @ 2024-03-10T15:50:40.958447+0000 1 0 2024-03-09T15:50:44.790 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T15:50:44.790 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T15:50:44.791 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:50:45.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:50:45.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:45.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:50:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:50:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:50:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:50:45.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:45.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T15:50:46.999 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:50:46.999 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:50:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:50:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:50:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:50:47.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:50:47.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:50:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:47.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:50:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:50:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:50:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:50:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:50:47.299 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:47.300+0000 7f0072b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:47.301 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:47.302+0000 7f0072b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:47.302 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:47.303+0000 7f0072b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:50:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:50:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:50:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:50:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:50:47.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:47.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:50:47.550 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:50:47.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:47.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:47.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:47.821+0000 7f0072b19640 -1 Falling back to public interface 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:49.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:50.044 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:50.044+0000 7f0072b19640 -1 osd.0 26 log_to_monitors true 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/554595198,v1:127.0.0.1:6803/554595198] [v2:127.0.0.1:6804/554595198,v1:127.0.0.1:6805/554595198] exists,up 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:50:50.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:50:50.430 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:50:50.430 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:50:50.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:50:50.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:50:50.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:50:50.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:50:50.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:50:50.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:50:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:50:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:50:50.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:50.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:50:50.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T15:50:50.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T15:50:50.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T15:50:50.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:50.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:50:51.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T15:50:51.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T15:50:51.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672975' 2024-03-09T15:50:51.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:50:51.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:50:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T15:50:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T15:50:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672975 2-64424509453' 2024-03-09T15:50:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:51.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T15:50:51.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:51.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:50:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T15:50:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T15:50:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T15:50:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 128849018882 2024-03-09T15:50:51.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:51.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T15:50:51.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:50:52.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:50:52.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:50:52.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T15:50:52.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:52.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T15:50:52.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:52.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:50:52.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T15:50:52.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:52.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T15:50:52.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T15:50:52.735 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672975 2024-03-09T15:50:52.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:50:53.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T15:50:53.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:50:53.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T15:50:53.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:50:53.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:50:53.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T15:50:53.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:50:53.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T15:50:53.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T15:50:53.007 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509453 2024-03-09T15:50:53.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:50:53.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-09T15:50:53.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:50:53.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:53.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:50:53.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:50:53.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:50:53.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T15:50:53.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:50:53.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:53.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:54.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T15:50:54.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T15:50:54.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:50:54.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T15:50:54.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:50:54.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:50:54.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:50:54.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:50:54.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:50:54.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:50:54.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:50:54.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:50:54.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:50:54.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:50:54.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:50:54.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:50:54.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:50:54.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:50:54.974 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T15:50:54.975 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T15:50:54.975 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:50:54.975 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:54.975 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:50:55.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T15:50:55.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T15:50:55.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T15:50:55.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:50:55.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:50:55.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:50:55.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:50:55.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:55.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:50:57.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:50:57.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T15:50:57.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T15:50:57.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T15:50:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T15:50:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:50:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:50:57.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:50:57.548 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:57.548+0000 7faad0222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:57.550 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:57.551+0000 7faad0222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:57.552 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:57.553+0000 7faad0222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:50:57.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:57.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:58.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:50:58.074 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:50:58.075+0000 7faad0222640 -1 Falling back to public interface 2024-03-09T15:50:59.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:50:59.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:50:59.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:50:59.069 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:50:59.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:50:59.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:50:59.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:00.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:00.360 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:00.361+0000 7faad0222640 -1 osd.0 31 log_to_monitors true 2024-03-09T15:51:00.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:01.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/3735243908,v1:127.0.0.1:6803/3735243908] [v2:127.0.0.1:6804/3735243908,v1:127.0.0.1:6805/3735243908] exists,up 5276e770-3ac0-4fc8-ab2c-37984e3f646a 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:51:01.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:51:01.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:51:02.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:51:02.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:51:02.294 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:51:02.294 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:51:02.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:51:02.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:51:02.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:51:02.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T15:51:02.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T15:51:02.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-09T15:51:02.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:51:02.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:51:02.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T15:51:02.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T15:51:02.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672978' 2024-03-09T15:51:02.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:51:02.501 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672978 2-64424509456' 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-09T15:51:02.629 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:51:02.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:51:02.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-09T15:51:02.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:51:02.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T15:51:02.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-09T15:51:02.632 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 150323855362 2024-03-09T15:51:02.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:51:02.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T15:51:02.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:51:02.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T15:51:02.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:51:02.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:51:02.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T15:51:02.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:51:02.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T15:51:02.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T15:51:02.905 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672978 2024-03-09T15:51:02.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:51:03.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-03-09T15:51:03.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:51:04.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:51:04.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:51:04.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-03-09T15:51:04.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:51:05.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:51:05.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:51:05.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-03-09T15:51:05.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:51:05.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T15:51:05.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:51:05.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:51:05.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T15:51:05.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:51:05.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T15:51:05.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T15:51:05.746 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509456 2024-03-09T15:51:05.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:51:06.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T15:51:06.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:51:06.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:51:06.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:51:06.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:51:06.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:51:06.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T15:51:06.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:51:06.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:51:06.634 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:51:06.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T15:51:06.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:51:06.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:51:06.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:51:06.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T15:51:06.971 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:50:40.957+0000 7f2199186640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T15:51:06.972 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:50:40.957+0000 7f2199186640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T15:51:06.972 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:50:40.957+0000 7f2199186640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T15:51:06.972 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:50:40.957+0000 7f2199186640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T15:51:06.972 INFO:tasks.workunit.client.0.smithi146.stdout:2024-03-09T15:50:40.957+0000 7f2199186640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:51:07.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:51:07.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:51:07.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:51:07.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:51:07.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:51:07.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:51:07.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:51:07.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:51:07.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:51:07.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:51:07.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:51:07.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:51:07.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:51:07.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:51:07.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:51:07.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:51:07.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:51:07.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:51:07.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:51:07.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:51:07.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:51:07.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:51:07.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:51:07.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:51:07.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:51:07.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:51:07.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:51:07.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:51:07.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:51:07.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:51:07.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:51:07.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:51:07.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:51:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:51:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:51:07.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:51:07.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:51:07.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:51:07.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:51:07.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:51:07.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:51:07.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:51:07.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:51:07.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:51:07.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:51:07.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:51:07.482 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:51:07.482 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:51:07.482 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:51:07.482 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:51:07.482 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.483 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:07.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:51:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:51:07.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:51:07.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:51:07.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:51:07.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:51:07.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:51:07.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:51:07.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:51:07.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:51:07.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:51:07.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:51:07.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:51:07.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:51:07.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:51:07.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:51:07.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:51:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:51:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:51:08.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:51:08.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:51:08.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:51:08.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:51:08.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:51:08.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6614bb1f-6867-4fa9-a164-f5ab860c684f 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6614bb1f-6867-4fa9-a164-f5ab860c684f' 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 6614bb1f-6867-4fa9-a164-f5ab860c684f 2024-03-09T15:51:08.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:51:08.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBshexlLnk7KRAAW/jiEnXFiKOpOPV5YC96EQ== 2024-03-09T15:51:08.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBshexlLnk7KRAAW/jiEnXFiKOpOPV5YC96EQ=="}' 2024-03-09T15:51:08.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6614bb1f-6867-4fa9-a164-f5ab860c684f -i td/osd-scrub-repair/0/new.json 2024-03-09T15:51:08.896 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:51:08.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:51:08.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBshexlLnk7KRAAW/jiEnXFiKOpOPV5YC96EQ== --osd-uuid 6614bb1f-6867-4fa9-a164-f5ab860c684f 2024-03-09T15:51:08.940 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.941+0000 7f44188b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:08.947 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.948+0000 7f44188b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:08.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.949+0000 7f44188b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:08.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.950+0000 7f44188b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:51:08.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.950+0000 7f44188b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:51:08.950 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:08.950+0000 7f44188b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:51:09.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:09.040+0000 7f44188b5640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:51:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:51:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:51:14.609 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:51:14.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:51:14.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:51:14.977 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:51:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:51:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:51:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:51:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:51:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:51:15.007 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:15.006+0000 7fb7db80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:15.020 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:15.019+0000 7fb7db80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:15.021 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:15.020+0000 7fb7db80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:15.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:51:15.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:51:15.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:51:15.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:15.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:15.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:15.565 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:15.565+0000 7fb7db80f640 -1 Falling back to public interface 2024-03-09T15:51:16.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:16.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:16.552 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:51:16.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:51:16.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:16.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:16.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:17.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:17.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:17.832 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:51:17.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:51:17.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:17.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:17.877 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:17.877+0000 7fb7db80f640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:51:18.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:19.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:19.129 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:51:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:51:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:19.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:20.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:20.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:20.433 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:51:20.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:51:20.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:20.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:20.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:21.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:21.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:21.818 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:51:21.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:51:21.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:21.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:22.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:23.097 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:51:23.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:51:23.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:23.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:23.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:24.431 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:51:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:51:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:24.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:24.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:25.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:25.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:25.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:51:25.831 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:51:25.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:25.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:26.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:27.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:27.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:27.113 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:51:27.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:51:27.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:27.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:27.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:28.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:28.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:28.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:51:28.437 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:51:28.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:28.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:28.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:29.804 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:30.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:31.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:31.250 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:51:31.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:31.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:51:31.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:31.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:31.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:32.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:32.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:32.733 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:51:32.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:51:32.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:32.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:33.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:34.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:34.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:34.392 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:51:34.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:51:34.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:34.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:34.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:35.854 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:51:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:51:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:36.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:37.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:37.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:37.406 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:51:37.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:51:37.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:37.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:37.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:38.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:38.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:38.740 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:51:38.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:51:38.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:38.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:39.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:40.299 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:51:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:51:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:51:40.595 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3367565372,v1:127.0.0.1:6803/3367565372] [v2:127.0.0.1:6804/3367565372,v1:127.0.0.1:6805/3367565372] exists,up 6614bb1f-6867-4fa9-a164-f5ab860c684f 2024-03-09T15:51:40.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:51:40.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:51:40.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:51:40.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:51:40.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:51:40.606 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:51:40.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:51:40.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:51:40.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:51:40.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:51:40.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:51:40.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:51:40.609 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:51:40.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16' 2024-03-09T15:51:40.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:51:40.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMhexl7L3CJBAAIsYfeWj0Z0bu/GhNKQQ9xQ== 2024-03-09T15:51:40.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMhexl7L3CJBAAIsYfeWj0Z0bu/GhNKQQ9xQ=="}' 2024-03-09T15:51:40.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 -i td/osd-scrub-repair/1/new.json 2024-03-09T15:51:40.967 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:51:40.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:51:40.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCMhexl7L3CJBAAIsYfeWj0Z0bu/GhNKQQ9xQ== --osd-uuid 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:51:41.011 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.011+0000 7fb704013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:41.035 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.035+0000 7fb704013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:41.036 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.036+0000 7fb704013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:41.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.037+0000 7fb704013640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:51:41.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.037+0000 7fb704013640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:51:41.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.037+0000 7fb704013640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:51:41.110 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:41.110+0000 7fb704013640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:51:47.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:51:47.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:51:47.350 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:51:47.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:51:47.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:51:47.705 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:51:47.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:51:47.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:51:47.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:51:47.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:51:47.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:51:47.735 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:47.736+0000 7fdf47062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:47.757 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:47.758+0000 7fdf47062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:47.758 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:47.759+0000 7fdf47062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:48.022 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:51:48.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:51:48.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:48.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:48.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:48.295 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:48.296+0000 7fdf47062640 -1 Falling back to public interface 2024-03-09T15:51:49.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:49.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:49.291 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:51:49.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:51:49.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:49.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:49.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:50.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:50.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:50.572 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:51:50.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:51:50.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:50.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:50.625 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:51:50.625+0000 7fdf47062640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:51:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:51.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:51.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:51.858 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:51:51.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:51:51.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:51.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:52.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:53.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:53.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:53.348 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:51:53.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:51:53.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:53.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:53.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:54.632 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:51:54.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:51:54.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:54.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:55.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:55.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:55.918 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:51:55.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:51:55.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:55.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:57.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:57.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:57.272 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:51:57.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:51:57.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:57.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:57.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:58.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:58.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:58.631 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:51:58.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:51:58.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:58.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:51:58.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:51:59.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:51:59.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:51:59.922 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:51:59.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:51:59.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:51:59.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:00.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:01.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:01.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:01.254 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:52:01.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:52:01.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:01.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:01.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:02.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:02.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:02.644 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:52:02.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:52:02.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:02.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:02.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:03.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:03.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:03.975 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:52:03.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:52:03.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:03.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:04.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:05.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:05.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:05.543 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:52:05.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:52:05.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:05.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:06.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:07.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:07.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:07.288 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:52:07.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:52:07.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:07.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:08.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:09.216 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:52:09.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:52:09.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:09.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:09.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:10.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:10.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:10.565 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:52:10.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:52:10.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:10.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:10.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:11.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:11.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:11.950 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:52:11.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:52:11.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:11.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:12.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:13.700 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:52:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:52:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:13.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:14.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:14.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:14.993 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:52:14.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:52:14.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:14.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:52:15.265 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/453467091,v1:127.0.0.1:6811/453467091] [v2:127.0.0.1:6812/453467091,v1:127.0.0.1:6813/453467091] exists,up 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:52:15.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:52:15.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:52:15.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:52:15.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:52:15.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=adc05be7-16ff-4bc0-9b6d-f27075e5423b 2024-03-09T15:52:15.273 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 adc05be7-16ff-4bc0-9b6d-f27075e5423b 2024-03-09T15:52:15.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 adc05be7-16ff-4bc0-9b6d-f27075e5423b' 2024-03-09T15:52:15.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:52:15.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvhexl+M8WERAAO8olVCxNEA7qZkj9VKlp7A== 2024-03-09T15:52:15.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvhexl+M8WERAAO8olVCxNEA7qZkj9VKlp7A=="}' 2024-03-09T15:52:15.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new adc05be7-16ff-4bc0-9b6d-f27075e5423b -i td/osd-scrub-repair/2/new.json 2024-03-09T15:52:15.639 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:52:15.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:52:15.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCvhexl+M8WERAAO8olVCxNEA7qZkj9VKlp7A== --osd-uuid adc05be7-16ff-4bc0-9b6d-f27075e5423b 2024-03-09T15:52:15.682 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.683+0000 7f4f14316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.724+0000 7f4f14316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.725+0000 7f4f14316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:15.725 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.726+0000 7f4f14316640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:52:15.725 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.726+0000 7f4f14316640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:52:15.725 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.726+0000 7f4f14316640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:52:15.799 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:15.799+0000 7f4f14316640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:52:22.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:52:22.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:52:22.267 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:52:22.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:52:22.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:52:22.621 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:52:22.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:52:22.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:52:22.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:52:22.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:52:22.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:52:22.650 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:22.650+0000 7f4141593640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:22.653 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:22.652+0000 7f4141593640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:22.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:22.653+0000 7f4141593640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:52:22.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:22.897 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:52:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:52:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:23.181 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:23.181+0000 7f4141593640 -1 Falling back to public interface 2024-03-09T15:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:24.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:24.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:24.243 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:52:24.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:52:24.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:24.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:24.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:25.495 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:52:25.494+0000 7f4141593640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:52:25.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:25.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:25.525 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:52:25.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:52:25.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:25.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:25.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:26.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:26.972 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:52:26.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:26.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:52:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:27.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:28.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:28.287 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:52:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:52:28.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:29.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:29.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:29.572 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:52:29.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:52:29.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:29.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:29.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:30.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:30.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:30.982 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:52:30.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:52:30.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:30.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:32.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:32.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:32.295 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:52:32.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:52:32.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:32.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:32.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:33.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:33.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:33.578 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:52:33.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:52:33.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:33.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:33.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:34.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:34.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:34.864 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:52:34.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:52:34.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:34.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:35.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:36.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:36.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:36.438 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:52:36.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:52:36.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:36.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:36.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:37.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:37.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:37.721 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:52:37.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:52:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:37.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:38.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:39.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:39.465 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:52:39.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:39.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:52:39.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:39.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:39.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:40.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:40.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:40.944 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:52:40.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:52:40.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:40.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:42.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:52:42.276 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:52:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:42.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:43.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:43.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:43.897 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:52:43.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:52:43.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:43.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:44.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:45.339 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:52:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:52:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:45.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:46.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:46.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:46.867 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:52:46.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:52:46.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:46.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:47.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:52:48.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:52:48.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:52:48.729 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:52:48.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:52:48.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:52:48.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:52:49.064 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1675524619,v1:127.0.0.1:6819/1675524619] [v2:127.0.0.1:6820/1675524619,v1:127.0.0.1:6821/1675524619] exists,up adc05be7-16ff-4bc0-9b6d-f27075e5423b 2024-03-09T15:52:49.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:52:49.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:52:49.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:52:49.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-09T15:52:49.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T15:52:49.333 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T15:52:49.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T15:52:49.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T15:52:49.713 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T15:52:49.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:52:50.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T15:52:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-09T15:52:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:52:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:52:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:52:52.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:52:52.701 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:52:52.701 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:52:52.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:52:52.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:52:52.702 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:52:52.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:52:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:52:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:52:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:52:52.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:52:52.797 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:52:53.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:52:53.068 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:52:53.069 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:52:53.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:52:53.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:53.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:52:53.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T15:52:53.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T15:52:53.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T15:52:53.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:53.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:52:53.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-09T15:52:53.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-09T15:52:53.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-64424509454' 2024-03-09T15:52:53.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:53.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:52:53.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-03-09T15:52:53.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-03-09T15:52:53.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-64424509454 2-128849018890' 2024-03-09T15:52:53.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:52:53.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T15:52:53.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:52:53.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:52:53.411 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T15:52:53.411 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:52:53.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T15:52:53.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T15:52:53.412 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836498 2024-03-09T15:52:53.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:52:53.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-09T15:52:53.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:52:54.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:52:54.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:52:54.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T15:52:54.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:52:54.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509454 2024-03-09T15:52:54.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:52:54.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:52:54.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509454 2024-03-09T15:52:54.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:52:54.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-09T15:52:54.965 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 64424509454 2024-03-09T15:52:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509454' 2024-03-09T15:52:54.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:52:55.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-09T15:52:55.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:52:55.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-128849018890 2024-03-09T15:52:55.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:52:55.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:52:55.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-128849018890 2024-03-09T15:52:55.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:52:55.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-03-09T15:52:55.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 128849018890' 2024-03-09T15:52:55.239 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 128849018890 2024-03-09T15:52:55.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:52:55.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018891 -lt 128849018890 2024-03-09T15:52:55.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:52:55.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:52:55.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:52:55.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:52:55.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:52:55.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:52:55.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:52:55.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:52:55.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:52:55.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:52:55.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:52:56.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:52:56.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:52:56.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:52:56.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T15:52:56.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T15:52:56.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T15:52:56.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T15:52:57.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T15:52:57.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T15:52:57.715 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T15:52:57.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:52:58.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T15:52:58.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T15:52:58.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:52:58.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:52:58.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:52:58.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:52:58.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:52:58.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:52:58.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:52:58.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:52:58.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:52:58.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:52:59.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:52:59.190 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:52:59.191 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:52:59.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:52:59.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:59.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:52:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T15:52:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T15:52:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T15:52:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:59.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:52:59.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T15:52:59.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T15:52:59.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-64424509456' 2024-03-09T15:52:59.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:52:59.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:52:59.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018893 2024-03-09T15:52:59.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018893 2024-03-09T15:52:59.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-64424509456 2-128849018893' 2024-03-09T15:52:59.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:52:59.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T15:52:59.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:52:59.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:52:59.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T15:52:59.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:52:59.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T15:52:59.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T15:52:59.551 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T15:52:59.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:52:59.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T15:52:59.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:53:00.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:53:00.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:01.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836500 2024-03-09T15:53:01.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:01.117 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509456 2024-03-09T15:53:01.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:01.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:53:01.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509456 2024-03-09T15:53:01.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:01.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T15:53:01.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509456' 2024-03-09T15:53:01.120 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 64424509456 2024-03-09T15:53:01.121 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:53:01.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-03-09T15:53:01.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:01.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-128849018893 2024-03-09T15:53:01.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:01.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:53:01.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-128849018893 2024-03-09T15:53:01.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:01.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018893 2024-03-09T15:53:01.404 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 128849018893 2024-03-09T15:53:01.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 128849018893' 2024-03-09T15:53:01.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:53:01.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018893 -lt 128849018893 2024-03-09T15:53:01.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:53:01.680 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:01.680 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:53:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:02.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:02.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:53:02.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:02.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:02.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:02.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T15:53:02.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T15:53:02.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:53:02.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T15:53:02.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:53:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:53:02.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:53:02.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:02.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:03.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:53:03.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:03.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:03.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:03.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T15:53:03.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T15:53:03.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:53:03.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:53:03.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:53:03.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:53:03.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:53:03.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:53:03.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:53:03.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:53:03.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:53:03.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T15:53:03.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:53:03.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T15:53:03.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:53:03.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:03.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:03.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:03.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:03.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:03.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:03.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:04.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:53:04.240 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:04.240 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:04.240 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:04.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:53:04.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:53:04.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:53:04.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T15:53:04.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-09T15:53:04.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-09T15:53:04.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:53:04.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T15:53:04.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T15:53:04.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:53:04.883 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T15:53:04.883 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T15:53:04.884 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:53:04.884 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:53:04.884 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:53:05.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T15:53:05.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:53:05.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T15:53:05.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:53:05.510 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:53:05.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:53:05.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:53:05.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:53:05.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:53:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:53:05.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:53:05.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T15:53:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T15:53:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T15:53:07.567 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T15:53:08.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:08.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:53:08.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:53:08.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:53:08.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:08.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:53:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:53:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:53:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:53:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:53:08.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:08.116+0000 7f95c9526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:08.118 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:08.119+0000 7f95c9526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:08.120 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:08.120+0000 7f95c9526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:08.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:53:08.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:53:08.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:53:08.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:08.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:08.647 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:08.648+0000 7f95c9526640 -1 Falling back to public interface 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:09.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:09.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:10.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:11.460 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:11.461+0000 7f95c9526640 -1 osd.1 52 log_to_monitors true 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:12.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:12.643 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 61 up_thru 61 down_at 55 last_clean_interval [15,52) [v2:127.0.0.1:6810/2559168612,v1:127.0.0.1:6811/2559168612] [v2:127.0.0.1:6812/2559168612,v1:127.0.0.1:6813/2559168612] exists,up 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:53:12.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:53:12.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:53:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:53:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:53:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:53:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:53:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:53:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:53:12.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:53:13.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:53:13.079 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:53:13.079 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:53:13.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:53:13.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:13.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:53:13.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-09T15:53:13.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-09T15:53:13.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-09T15:53:13.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:13.318 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:53:13.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005058 2024-03-09T15:53:13.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005058 2024-03-09T15:53:13.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-261993005058' 2024-03-09T15:53:13.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:13.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018896 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018896 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-261993005058 2-128849018896' 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-09T15:53:13.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:13.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:53:13.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-09T15:53:13.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:13.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-09T15:53:13.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-09T15:53:13.551 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836504 2024-03-09T15:53:13.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:13.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-09T15:53:13.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:53:14.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:53:14.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:15.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-09T15:53:15.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:15.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005058 2024-03-09T15:53:15.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:15.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:53:15.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005058 2024-03-09T15:53:15.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:15.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005058 2024-03-09T15:53:15.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005058' 2024-03-09T15:53:15.139 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 261993005058 2024-03-09T15:53:15.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:53:15.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005058 -lt 261993005058 2024-03-09T15:53:15.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:15.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-128849018896 2024-03-09T15:53:15.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:15.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:53:15.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-128849018896 2024-03-09T15:53:15.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:15.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018896 2024-03-09T15:53:15.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 128849018896' 2024-03-09T15:53:15.415 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 128849018896 2024-03-09T15:53:15.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:53:15.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018897 -lt 128849018896 2024-03-09T15:53:15.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:53:15.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:15.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:16.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:16.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:53:16.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:16.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:16.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:53:16.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:53:16.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:53:16.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-09T15:53:16.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-09T15:53:16.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:53:16.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:53:16.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:53:16.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:53:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T15:53:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:53:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:53:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:53:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:53:03.748082+0000 '>' 2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:53:18.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:53:18.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:53:18.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:53:18.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:53:18.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:53:18.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:53:18.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:53:18.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:53:03.748082+0000 '>' 2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:18.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:53:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:53:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:53:19.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:53:19.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:53:19.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:53:19.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:53:19.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:53:20.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:53:03.748082+0000 '>' 2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:20.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:53:21.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:53:21.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:53:21.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:53:21.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:53:21.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:53:21.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:53:21.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:53:15.648251+0000 '>' 2024-03-09T15:53:03.748082+0000 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:53:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:53:21.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:53:21.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:53:21.725 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:53:21.726 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:53:21.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:53:21.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:21.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:53:21.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-09T15:53:21.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-09T15:53:21.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-09T15:53:21.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:21.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:53:21.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005061 2024-03-09T15:53:21.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005061 2024-03-09T15:53:21.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-261993005061' 2024-03-09T15:53:21.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:21.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:53:22.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018899 2024-03-09T15:53:22.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018899 2024-03-09T15:53:22.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-261993005061 2-128849018899' 2024-03-09T15:53:22.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:22.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-09T15:53:22.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:22.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:53:22.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-09T15:53:22.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:22.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-09T15:53:22.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-09T15:53:22.077 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836507 2024-03-09T15:53:22.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:22.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-09T15:53:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:53:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:53:23.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:23.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-09T15:53:23.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:23.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005061 2024-03-09T15:53:23.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:23.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:53:23.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005061 2024-03-09T15:53:23.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:23.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005061 2024-03-09T15:53:23.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005061' 2024-03-09T15:53:23.645 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 261993005061 2024-03-09T15:53:23.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:53:23.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005061 -lt 261993005061 2024-03-09T15:53:23.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:23.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-128849018899 2024-03-09T15:53:23.921 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:23.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:53:23.921 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-128849018899 2024-03-09T15:53:23.921 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:23.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018899 2024-03-09T15:53:23.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 128849018899' 2024-03-09T15:53:23.923 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 128849018899 2024-03-09T15:53:23.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:53:24.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018899 -lt 128849018899 2024-03-09T15:53:24.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:53:24.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:24.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:24.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:53:24.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:24.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:24.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:24.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:24.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:24.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:24.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:24.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:53:24.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:24.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:24.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:25.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:53:25.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:53:25.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:53:25.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T15:53:25.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:53:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:53:25.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T15:53:25.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:53:25.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:53:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:53:25.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:53:25.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:53:25.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:53:25.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:53:25.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:53:26.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:53:26.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:53:26.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:53:26.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:53:26.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:53:26.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:53:26.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:53:26.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T15:53:28.020 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T15:53:28.021 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T15:53:28.021 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:53:28.021 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T15:53:28.021 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:53:28.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:28.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:28.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:53:28.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:53:28.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:53:28.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:53:28.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:53:28.321 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:28.321+0000 7fc1544f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:28.323 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:28.323+0000 7fc1544f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:28.324 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:28.325+0000 7fc1544f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:28.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:28.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:28.851 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:28.852+0000 7fc1544f9640 -1 Falling back to public interface 2024-03-09T15:53:29.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:29.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:29.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:53:29.842 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:53:29.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:29.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:30.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:31.234 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:31.234+0000 7fc1544f9640 -1 osd.1 70 log_to_monitors true 2024-03-09T15:53:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:32.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 71 last_clean_interval [61,70) [v2:127.0.0.1:6810/166843188,v1:127.0.0.1:6811/166843188] [v2:127.0.0.1:6812/166843188,v1:127.0.0.1:6813/166843188] exists,up 448ba0c9-2c46-49d4-a835-1dc8b0a3fd16 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:53:32.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:53:32.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:53:32.989 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:53:33.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:53:33.364 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:53:33.364 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:53:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:53:33.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:33.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:53:33.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-09T15:53:33.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-09T15:53:33.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-09T15:53:33.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:33.489 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:53:33.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T15:53:33.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T15:53:33.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-330712481794' 2024-03-09T15:53:33.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:53:33.597 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018902 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018902 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-330712481794 2-128849018902' 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-09T15:53:33.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:33.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:53:33.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-09T15:53:33.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:33.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-09T15:53:33.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-09T15:53:33.722 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836510 2024-03-09T15:53:33.722 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:33.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-03-09T15:53:33.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:53:34.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:53:34.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:53:35.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-09T15:53:35.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:35.277 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-09T15:53:35.277 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:35.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:53:35.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-09T15:53:35.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:35.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T15:53:35.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-09T15:53:35.279 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 330712481794 2024-03-09T15:53:35.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:53:35.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-09T15:53:35.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:53:35.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-128849018902 2024-03-09T15:53:35.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:53:35.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:53:35.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-128849018902 2024-03-09T15:53:35.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:53:35.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018902 2024-03-09T15:53:35.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 128849018902' 2024-03-09T15:53:35.557 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 128849018902 2024-03-09T15:53:35.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:53:35.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018903 -lt 128849018902 2024-03-09T15:53:35.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:53:35.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:35.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:53:36.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:53:36.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:53:36.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:53:36.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:53:36.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:53:36.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:53:36.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:53:36.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:53:36.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:53:36.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:53:36.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:53:36.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:53:36.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:53:36.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:53:36.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:53:36.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:53:36.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:53:36.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:53:36.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:53:37.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:53:37.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:53:37.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:53:37.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:53:37.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:53:37.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:53:37.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:53:37.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:53:37.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:53:37.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:53:37.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:53:37.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:53:37.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:53:37.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:53:37.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:53:37.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:53:37.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:53:37.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:53:37.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:53:37.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:53:37.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:53:37.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:53:37.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:53:37.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:53:37.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:53:37.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:53:37.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:53:37.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:53:37.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:53:37.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:53:37.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:53:37.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:53:37.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:53:37.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:53:37.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:53:37.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:53:37.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:53:37.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:53:37.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:53:37.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:53:37.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-03-09T15:53:37.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T15:53:37.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:53:37.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:53:37.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:53:37.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:53:37.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:53:37.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:37.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:53:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:53:37.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:53:37.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:53:37.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:53:37.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:53:37.689 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:53:37.690 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:53:37.690 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:53:37.690 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:53:37.690 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:53:37.691 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.691 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:53:37.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:53:37.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:53:37.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:53:37.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:53:37.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:53:38.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:38.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:53:38.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:53:38.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:53:38.695 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-09T15:53:38.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:53:38.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:38.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:53:38.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:53:38.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:53:38.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:53:38.702 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:53:38.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82134875-b697-43df-8fa4-03794882e38d 2024-03-09T15:53:38.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 82134875-b697-43df-8fa4-03794882e38d' 2024-03-09T15:53:38.702 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 82134875-b697-43df-8fa4-03794882e38d 2024-03-09T15:53:38.703 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:53:38.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAChuxlZ7XYKhAA5UWR52ApJn0WkEMzSokQUA== 2024-03-09T15:53:38.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAChuxlZ7XYKhAA5UWR52ApJn0WkEMzSokQUA=="}' 2024-03-09T15:53:38.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82134875-b697-43df-8fa4-03794882e38d -i td/osd-scrub-repair/0/new.json 2024-03-09T15:53:38.920 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:53:38.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:53:38.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAChuxlZ7XYKhAA5UWR52ApJn0WkEMzSokQUA== --osd-uuid 82134875-b697-43df-8fa4-03794882e38d 2024-03-09T15:53:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.962+0000 7f6869648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:38.971 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.970+0000 7f6869648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:38.972 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.971+0000 7f6869648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:38.972 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.972+0000 7f6869648640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:53:38.972 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.972+0000 7f6869648640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:53:38.972 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:38.972+0000 7f6869648640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:53:39.063 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:39.062+0000 7f6869648640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:53:44.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:53:44.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:53:44.640 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:53:44.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:53:44.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:53:44.994 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:53:44.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:53:44.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:53:44.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:53:44.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:53:44.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:53:45.024 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:45.025+0000 7f18b500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:45.026 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:45.027+0000 7f18b500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:45.027 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:45.028+0000 7f18b500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:53:45.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:53:45.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:53:45.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:53:45.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:45.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:45.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:45.561 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:45.562+0000 7f18b500f640 -1 Falling back to public interface 2024-03-09T15:53:46.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:46.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:46.539 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:53:46.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:53:46.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:46.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:46.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:47.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:47.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:47.819 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:53:47.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:53:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:47.903 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:53:47.904+0000 7f18b500f640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:53:48.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:49.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:49.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:49.097 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:53:49.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:53:49.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:49.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:50.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:50.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:50.379 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:53:50.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:53:50.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:50.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:50.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:51.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:51.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:51.664 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:53:51.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:53:51.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:51.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:51.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:52.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:52.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:52.946 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:53:52.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:53:52.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:52.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:53.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:54.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:54.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:54.231 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:53:54.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:53:54.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:54.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:54.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:55.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:55.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:55.516 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:53:55.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:53:55.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:55.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:55.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:56.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:56.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:56.847 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:53:56.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:53:56.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:56.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:57.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:58.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:58.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:58.127 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:53:58.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:53:58.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:58.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:53:58.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:53:59.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:53:59.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:53:59.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:53:59.685 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:53:59.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:53:59.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:00.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:01.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:01.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:01.172 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:54:01.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:54:01.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:01.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:01.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:02.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:02.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:02.571 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:54:02.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:54:02.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:02.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:03.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:04.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:04.391 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:54:04.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:54:04.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:04.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:04.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:05.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:05.825 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:54:05.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:05.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:54:05.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:05.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:07.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:07.539 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:54:07.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:07.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:54:07.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:07.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:08.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:09.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:09.214 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:54:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:54:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:09.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:54:09.581 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2577199857,v1:127.0.0.1:6803/2577199857] [v2:127.0.0.1:6804/2577199857,v1:127.0.0.1:6805/2577199857] exists,up 82134875-b697-43df-8fa4-03794882e38d 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:54:09.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:54:09.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:54:09.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:54:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1d1ed235-f228-46e9-9f96-11bfdc20adef' 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:54:09.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:54:09.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhhuxlMYjvIxAAoxft68dhMKsws4MlVTOEfA== 2024-03-09T15:54:09.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhhuxlMYjvIxAAoxft68dhMKsws4MlVTOEfA=="}' 2024-03-09T15:54:09.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d1ed235-f228-46e9-9f96-11bfdc20adef -i td/osd-scrub-repair/1/new.json 2024-03-09T15:54:09.922 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:54:09.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:54:09.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAhhuxlMYjvIxAAoxft68dhMKsws4MlVTOEfA== --osd-uuid 1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:54:09.966 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:09.967+0000 7f348fe21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:10.007 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.007+0000 7f348fe21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:10.008 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.008+0000 7f348fe21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:10.008 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.009+0000 7f348fe21640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:54:10.008 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.009+0000 7f348fe21640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:54:10.008 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.009+0000 7f348fe21640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:54:10.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:10.082+0000 7f348fe21640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:54:16.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:54:16.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:54:16.315 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:54:16.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:54:16.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:54:16.666 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:54:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:54:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:54:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:54:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:54:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:54:16.696 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:16.696+0000 7fce26668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:16.697 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:16.698+0000 7fce26668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:16.698 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:16.699+0000 7fce26668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:54:16.943 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:54:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:17.218 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:17.219+0000 7fce26668640 -1 Falling back to public interface 2024-03-09T15:54:17.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:18.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:18.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:18.245 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:54:18.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:54:18.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:18.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:18.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:19.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:19.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:19.528 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:54:19.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:54:19.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:19.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:19.757 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:19.758+0000 7fce26668640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:54:19.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:20.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:20.822 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:54:20.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:20.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:54:20.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:20.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:21.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:22.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:22.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:22.163 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:54:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:54:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:22.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:23.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:23.501 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:54:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:54:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:23.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:24.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:24.785 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:54:24.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:24.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:54:24.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:24.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:25.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:26.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:26.104 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:54:26.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:26.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:54:26.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:26.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:26.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:27.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:27.487 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:54:27.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:27.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:54:27.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:27.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:27.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:28.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:28.876 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:54:28.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:28.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:54:28.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:28.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:30.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:30.161 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:54:30.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:30.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:54:30.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:30.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:30.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:31.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:31.573 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:54:31.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:31.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:54:31.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:31.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:31.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:32.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:32.854 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:54:32.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:32.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:54:32.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:32.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:33.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:34.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:34.730 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:54:34.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:34.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:54:34.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:34.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:35.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:36.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:36.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:36.191 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:54:36.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:54:36.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:36.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:36.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:37.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:37.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:37.652 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:54:37.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:54:37.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:37.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:38.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:39.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:39.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:39.205 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:54:39.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:54:39.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:39.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:39.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:40.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:40.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:40.907 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:54:40.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:54:40.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:40.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:41.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:42.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:42.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:42.701 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:54:42.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:54:42.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:42.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:43.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:44.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:44.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:44.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:54:44.046 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:54:44.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:44.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:54:44.318 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2515906105,v1:127.0.0.1:6811/2515906105] [v2:127.0.0.1:6812/2515906105,v1:127.0.0.1:6813/2515906105] exists,up 1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:54:44.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:54:44.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:54:44.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:54:44.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:54:44.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41b07fd6-246a-4f7e-b130-fcb6ecc9f036 2024-03-09T15:54:44.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 41b07fd6-246a-4f7e-b130-fcb6ecc9f036' 2024-03-09T15:54:44.326 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 41b07fd6-246a-4f7e-b130-fcb6ecc9f036 2024-03-09T15:54:44.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:54:44.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEhuxlNX9GFBAANIz0vabbfkaXMu7ChVriuQ== 2024-03-09T15:54:44.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEhuxlNX9GFBAANIz0vabbfkaXMu7ChVriuQ=="}' 2024-03-09T15:54:44.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41b07fd6-246a-4f7e-b130-fcb6ecc9f036 -i td/osd-scrub-repair/2/new.json 2024-03-09T15:54:44.702 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:54:44.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:54:44.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBEhuxlNX9GFBAANIz0vabbfkaXMu7ChVriuQ== --osd-uuid 41b07fd6-246a-4f7e-b130-fcb6ecc9f036 2024-03-09T15:54:44.745 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.745+0000 7f0481213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:44.787 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.786+0000 7f0481213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:44.788 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.787+0000 7f0481213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:44.788 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.788+0000 7f0481213640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:54:44.788 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.788+0000 7f0481213640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:54:44.788 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.788+0000 7f0481213640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:54:44.858 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:44.858+0000 7f0481213640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:54:51.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:54:51.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:54:51.155 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:54:51.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:54:51.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:54:51.509 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:54:51.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:54:51.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:54:51.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:54:51.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:54:51.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:54:51.539 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:51.539+0000 7efdc7099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:51.541 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:51.541+0000 7efdc7099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:51.541+0000 7efdc7099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:54:51.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:51.782 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:54:51.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:54:51.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:51.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:52.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:52.066 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:52.066+0000 7efdc7099640 -1 Falling back to public interface 2024-03-09T15:54:53.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:53.056 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:54:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:54:53.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:53.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:53.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:54.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:54.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:54.333 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:54:54.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:54:54.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:54.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:54.414 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:54:54.414+0000 7efdc7099640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:54:54.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:55.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:55.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:55.640 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:54:55.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:54:55.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:55.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:55.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:56.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:56.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:56.922 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:54:56.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:54:56.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:56.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:57.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:58.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:58.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:58.315 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:54:58.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:54:58.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:58.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:54:58.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:54:59.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:54:59.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:54:59.656 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:54:59.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:54:59.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:54:59.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:00.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:01.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:01.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:01.038 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:55:01.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:55:01.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:01.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:01.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:02.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:02.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:02.351 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:55:02.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:55:02.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:02.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:02.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:03.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:03.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:03.632 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:55:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:55:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:04.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:05.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:05.002 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:55:05.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:05.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:55:05.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:05.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:05.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:06.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:06.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:06.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:55:06.481 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:55:06.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:06.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:06.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:07.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:07.788 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:55:07.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:07.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:55:07.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:07.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:08.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:09.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:09.300 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:55:09.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:09.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:55:09.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:09.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:10.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:11.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:11.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:11.070 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:55:11.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:55:11.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:11.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:11.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:12.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:12.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:12.544 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:55:12.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:55:12.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:12.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:13.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:14.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:14.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:14.225 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:55:14.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:55:14.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:14.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:14.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:15.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:15.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:15.955 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:55:15.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:55:15.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:15.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:16.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:17.755 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:55:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:55:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:18.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:19.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:19.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:19.273 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:55:19.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:55:19.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:19.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:55:19.576 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1723662439,v1:127.0.0.1:6819/1723662439] [v2:127.0.0.1:6820/1723662439,v1:127.0.0.1:6821/1723662439] exists,up 41b07fd6-246a-4f7e-b130-fcb6ecc9f036 2024-03-09T15:55:19.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:55:19.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:55:19.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:55:19.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-09T15:55:19.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T15:55:19.842 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T15:55:19.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T15:55:19.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T15:55:20.226 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T15:55:20.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:55:21.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T15:55:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-09T15:55:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:55:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:55:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:55:23.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:55:23.440 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:55:23.440 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:55:23.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:55:23.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:55:23.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:55:23.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:55:23.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:55:23.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:55:23.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:55:23.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:55:23.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:55:23.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:55:23.809 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:55:23.809 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:55:23.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:55:23.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:23.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:55:23.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T15:55:23.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T15:55:23.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T15:55:23.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:23.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:55:24.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476748 2024-03-09T15:55:24.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476748 2024-03-09T15:55:24.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476748' 2024-03-09T15:55:24.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:24.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:55:24.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953479 2024-03-09T15:55:24.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953479 2024-03-09T15:55:24.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476748 2-137438953479' 2024-03-09T15:55:24.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:24.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T15:55:24.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:24.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:55:24.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T15:55:24.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:24.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T15:55:24.121 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T15:55:24.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T15:55:24.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:24.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T15:55:24.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:55:25.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:55:25.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:25.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T15:55:25.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:25.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476748 2024-03-09T15:55:25.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:25.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:55:25.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476748 2024-03-09T15:55:25.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476748 2024-03-09T15:55:25.708 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 68719476748 2024-03-09T15:55:25.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476748' 2024-03-09T15:55:25.708 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:26.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476748 -lt 68719476748 2024-03-09T15:55:26.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:26.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953479 2024-03-09T15:55:26.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:26.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:55:26.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953479 2024-03-09T15:55:26.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:26.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953479 2024-03-09T15:55:26.005 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 137438953479 2024-03-09T15:55:26.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953479' 2024-03-09T15:55:26.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:55:26.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953479 -lt 137438953479 2024-03-09T15:55:26.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:55:26.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:26.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:26.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:55:26.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:55:26.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:55:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:55:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:26.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:27.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T15:55:27.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T15:55:27.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T15:55:27.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T15:55:28.450 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T15:55:28.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:55:29.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T15:55:29.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T15:55:30.008 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:55:30.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:55:30.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:55:30.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:55:30.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:55:30.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:55:30.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:55:30.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:55:30.107 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:55:30.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:55:30.464 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:55:30.464 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:55:30.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:55:30.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:30.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:55:30.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T15:55:30.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T15:55:30.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T15:55:30.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:30.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:55:30.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476750 2024-03-09T15:55:30.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476750 2024-03-09T15:55:30.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-68719476750' 2024-03-09T15:55:30.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:30.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:55:30.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953481 2024-03-09T15:55:30.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953481 2024-03-09T15:55:30.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-68719476750 2-137438953481' 2024-03-09T15:55:30.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:30.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T15:55:30.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:30.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:55:30.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T15:55:30.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:30.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T15:55:30.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T15:55:30.808 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836503 2024-03-09T15:55:30.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:31.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T15:55:31.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:31.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476750 2024-03-09T15:55:31.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:31.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:55:31.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476750 2024-03-09T15:55:31.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:31.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476750 2024-03-09T15:55:31.103 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 68719476750 2024-03-09T15:55:31.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-03-09T15:55:31.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:31.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476749 -lt 68719476750 2024-03-09T15:55:31.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:55:32.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:55:32.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:32.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476749 -lt 68719476750 2024-03-09T15:55:32.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:55:33.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T15:55:33.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:33.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476750 -lt 68719476750 2024-03-09T15:55:33.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:33.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953481 2024-03-09T15:55:33.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:33.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:55:33.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953481 2024-03-09T15:55:33.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:33.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953481 2024-03-09T15:55:33.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953481' 2024-03-09T15:55:33.995 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 137438953481 2024-03-09T15:55:33.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:55:34.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953481 -lt 137438953481 2024-03-09T15:55:34.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:55:34.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:34.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:55:34.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:55:34.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:55:34.895 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:55:34.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:34.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-09T15:55:35.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:55:35.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:55:35.545 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:55:35.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T15:55:35.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:55:35.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:55:36.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:55:36.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:55:36.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:55:36.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:55:36.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:55:36.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:55:36.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:55:36.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:55:36.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:55:36.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T15:55:38.140 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T15:55:38.140 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T15:55:38.141 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T15:55:38.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:55:38.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:55:38.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:55:38.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:55:38.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:55:38.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:55:38.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:55:38.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:55:38.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:55:38.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:55:38.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:55:38.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:55:38.690 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:38.690+0000 7f720eb81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:38.692 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:38.692+0000 7f720eb81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:38.693 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:38.694+0000 7f720eb81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:38.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:55:39.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:39.207+0000 7f720eb81640 -1 Falling back to public interface 2024-03-09T15:55:39.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:40.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:55:40.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:41.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:41.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:41.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:55:41.572 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:55:41.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:41.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:55:41.670 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:41.670+0000 7f720eb81640 -1 osd.1 54 log_to_monitors true 2024-03-09T15:55:41.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:55:42.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:55:42.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:42.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:55:42.885 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:55:42.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:42.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 55 last_clean_interval [16,54) [v2:127.0.0.1:6810/3610423452,v1:127.0.0.1:6811/3610423452] [v2:127.0.0.1:6812/3610423452,v1:127.0.0.1:6813/3610423452] exists,up 1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:55:43.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:55:43.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:55:43.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:55:43.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:55:43.563 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:55:43.563 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:55:43.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:55:43.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:43.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:55:43.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-09T15:55:43.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-09T15:55:43.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-09T15:55:43.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:43.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:55:43.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-03-09T15:55:43.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-03-09T15:55:43.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-257698037762' 2024-03-09T15:55:43.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:43.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953484 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953484 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-257698037762 2-137438953484' 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-09T15:55:43.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:43.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:55:43.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-09T15:55:43.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:43.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-09T15:55:43.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-09T15:55:43.928 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836506 2024-03-09T15:55:43.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:44.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-03-09T15:55:44.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:55:45.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:55:45.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:45.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-09T15:55:45.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:45.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037762 2024-03-09T15:55:45.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:45.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:55:45.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037762 2024-03-09T15:55:45.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:45.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-03-09T15:55:45.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-03-09T15:55:45.495 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 257698037762 2024-03-09T15:55:45.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:45.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-03-09T15:55:45.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:45.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953484 2024-03-09T15:55:45.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:45.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:55:45.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953484 2024-03-09T15:55:45.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:45.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953484 2024-03-09T15:55:45.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953484' 2024-03-09T15:55:45.810 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 137438953484 2024-03-09T15:55:45.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:55:46.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953485 -lt 137438953484 2024-03-09T15:55:46.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:55:46.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:46.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:55:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:55:46.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:55:46.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:55:46.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:55:46.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:55:46.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:46.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:55:47.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:55:47.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:55:47.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-09T15:55:47.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-09T15:55:47.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:55:47.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:55:47.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:55:47.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:55:47.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:55:47.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:55:34.572200+0000 '>' 2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:47.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:55:48.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:55:48.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:55:48.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:55:48.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:55:48.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:55:48.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:55:48.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:55:49.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:55:34.572200+0000 '>' 2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:49.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:55:50.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:55:50.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:55:50.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:55:50.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:55:50.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:55:50.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:55:50.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:55:50.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:55:34.572200+0000 '>' 2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:50.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:55:51.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:55:51.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:55:51.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:55:51.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:55:51.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:55:51.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:55:51.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:55:51.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:55:47.611760+0000 '>' 2024-03-09T15:55:34.572200+0000 2024-03-09T15:55:51.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:55:51.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:55:51.875 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:55:52.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:55:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:55:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:55:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:55:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:52.148 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:55:52.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T15:55:52.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T15:55:52.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T15:55:52.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:52.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:55:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-03-09T15:55:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-03-09T15:55:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-257698037765' 2024-03-09T15:55:52.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:55:52.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953487 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953487 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-257698037765 2-137438953487' 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:52.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:55:52.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T15:55:52.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:52.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T15:55:52.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T15:55:52.486 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836509 2024-03-09T15:55:52.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:52.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T15:55:52.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:55:53.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:55:53.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:55:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-09T15:55:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:54.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037765 2024-03-09T15:55:54.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:55:54.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037765 2024-03-09T15:55:54.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:54.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-03-09T15:55:54.083 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 257698037765 2024-03-09T15:55:54.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037765' 2024-03-09T15:55:54.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:55:54.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037765 -lt 257698037765 2024-03-09T15:55:54.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:55:54.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953487 2024-03-09T15:55:54.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:55:54.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:55:54.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953487 2024-03-09T15:55:54.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:55:54.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953487 2024-03-09T15:55:54.366 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 137438953487 2024-03-09T15:55:54.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953487' 2024-03-09T15:55:54.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:55:54.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953487 -lt 137438953487 2024-03-09T15:55:54.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:55:54.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:54.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:55:54.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:55:55.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:55:55.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:55:55.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:55:55.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:55:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:55:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:55:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:55:55.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T15:55:55.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:55:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:55:55.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T15:55:55.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:55:55.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T15:55:56.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:55:56.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:55:56.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:55:56.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:55:56.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T15:55:58.180 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T15:55:58.180 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:55:58.180 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T15:55:58.180 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:55:58.181 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:55:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:55:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:55:58.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T15:55:58.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:55:58.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:55:58.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:55:58.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:55:58.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:55:58.489 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:58.488+0000 7f20a3ebf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:58.515 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:58.515+0000 7f20a3ebf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:58.517 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:58.516+0000 7f20a3ebf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:55:58.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:55:58.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:55:59.050 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:55:59.050+0000 7f20a3ebf640 -1 Falling back to public interface 2024-03-09T15:55:59.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:00.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:56:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:56:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:00.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:00.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:01.341 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:01.341+0000 7f20a3ebf640 -1 osd.1 68 log_to_monitors true 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:01.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:01.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:02.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:02.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:02.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:56:02.751 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:56:02.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:02.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 69 last_clean_interval [60,68) [v2:127.0.0.1:6810/2579834725,v1:127.0.0.1:6811/2579834725] [v2:127.0.0.1:6812/2579834725,v1:127.0.0.1:6813/2579834725] exists,up 1d1ed235-f228-46e9-9f96-11bfdc20adef 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:56:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:56:03.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:56:03.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:56:03.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:56:03.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-09T15:56:03.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-09T15:56:03.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-09T15:56:03.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:56:03.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:56:03.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-09T15:56:03.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-09T15:56:03.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-322122547202' 2024-03-09T15:56:03.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:56:03.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953491 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953491 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-322122547202 2-137438953491' 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-09T15:56:04.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:56:04.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:56:04.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-09T15:56:04.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:56:04.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-09T15:56:04.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-09T15:56:04.081 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836512 2024-03-09T15:56:04.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:56:04.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836512 2024-03-09T15:56:04.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:56:05.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:56:05.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:56:05.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-09T15:56:05.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:56:05.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-03-09T15:56:05.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:56:05.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:56:05.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-03-09T15:56:05.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:56:05.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-09T15:56:05.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-03-09T15:56:05.638 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 322122547202 2024-03-09T15:56:05.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:56:05.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547202 -lt 322122547202 2024-03-09T15:56:05.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:56:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953491 2024-03-09T15:56:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:56:05.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:56:05.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953491 2024-03-09T15:56:05.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:56:05.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953491 2024-03-09T15:56:05.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953491' 2024-03-09T15:56:05.910 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 137438953491 2024-03-09T15:56:05.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:56:06.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953491 -lt 137438953491 2024-03-09T15:56:06.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:56:06.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:56:06.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:56:06.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:56:06.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:56:06.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:56:06.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:56:06.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:56:07.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:56:07.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:56:07.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:56:07.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:56:07.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:56:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:56:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:56:07.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:56:07.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:56:07.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:56:07.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:56:07.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:56:07.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:56:07.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:56:07.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:56:07.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:56:07.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:56:07.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:56:07.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:56:07.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:56:07.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:56:07.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:07.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:07.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:56:07.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:56:07.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:56:07.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T15:56:07.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T15:56:07.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T15:56:07.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T15:56:07.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T15:56:07.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T15:56:07.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:56:07.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T15:56:07.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T15:56:07.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T15:56:07.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T15:56:07.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T15:56:07.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T15:56:07.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T15:56:07.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:07.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:07.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T15:56:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T15:56:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T15:56:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T15:56:07.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T15:56:07.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:07.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T15:56:07.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T15:56:07.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T15:56:07.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T15:56:07.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T15:56:07.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T15:56:07.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T15:56:07.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:07.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T15:56:08.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:56:08.048 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:56:08.049 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:56:08.049 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:08.049 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:08.050 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:56:08.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:56:08.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T15:56:08.130 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T15:56:08.131 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T15:56:08.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:09.023 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:09.024 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:56:09.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T15:56:09.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:56:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:56:09.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb 2024-03-09T15:56:09.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb' 2024-03-09T15:56:09.059 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb 2024-03-09T15:56:09.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:56:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZhuxlHJttBBAAPNEzJgYbu/DWGRPJOUGYpg== 2024-03-09T15:56:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZhuxlHJttBBAAPNEzJgYbu/DWGRPJOUGYpg=="}' 2024-03-09T15:56:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb -i td/osd-scrub-repair/0/new.json 2024-03-09T15:56:09.279 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:56:09.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T15:56:09.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZhuxlHJttBBAAPNEzJgYbu/DWGRPJOUGYpg== --osd-uuid d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb 2024-03-09T15:56:09.318 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.318+0000 7fcd011e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.330+0000 7fcd011e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:09.330 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.331+0000 7fcd011e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:09.331 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.331+0000 7fcd011e1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:56:09.331 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.331+0000 7fcd011e1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:56:09.331 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.332+0000 7fcd011e1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T15:56:09.421 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:09.422+0000 7fcd011e1640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T15:56:14.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T15:56:14.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:56:14.951 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T15:56:14.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T15:56:14.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:56:15.313 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T15:56:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T15:56:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:56:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:56:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:56:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:56:15.339 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:15.340+0000 7f0630136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:15.341 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:15.342+0000 7f0630136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:15.342 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:15.343+0000 7f0630136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:56:15.588 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:56:15.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:15.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:15.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:15.871+0000 7f0630136640 -1 Falling back to public interface 2024-03-09T15:56:16.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:16.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:16.860 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:56:16.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:56:16.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:16.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:17.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:18.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:18.139 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:56:18.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:18.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:56:18.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:18.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:18.200 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:18.200+0000 7f0630136640 -1 osd.0 0 log_to_monitors true 2024-03-09T15:56:18.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:19.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:19.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:19.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:56:19.444 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:56:19.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:19.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:19.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:20.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:20.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:20.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:56:20.739 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:56:20.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:20.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:21.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:22.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:22.021 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:56:22.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:22.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:56:22.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:22.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:22.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:23.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:23.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:23.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:56:23.304 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:56:23.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:23.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:23.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:24.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:24.586 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:56:24.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:24.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:56:24.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:24.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:24.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:25.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:25.986 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:56:25.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:25.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:56:25.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:25.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:26.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:27.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:27.269 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:56:27.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:27.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:56:27.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:27.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:27.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:28.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:28.551 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:56:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:56:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:28.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:29.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:29.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:29.948 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:56:29.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:56:29.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:29.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:30.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:31.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:31.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:31.430 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:56:31.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:56:31.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:31.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:31.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:32.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:32.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:32.979 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:56:32.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:56:32.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:32.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:33.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:34.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:34.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:34.613 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:56:34.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:56:34.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:34.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:35.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:36.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:36.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:36.406 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:56:36.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:56:36.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:36.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:37.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:38.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:38.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:38.142 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:56:38.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:56:38.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:38.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:38.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:39.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:39.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:39.969 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:56:39.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:56:39.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:39.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:40.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:41.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:41.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:41.313 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:56:41.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:56:41.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:41.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T15:56:41.635 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3169280041,v1:127.0.0.1:6803/3169280041] [v2:127.0.0.1:6804/3169280041,v1:127.0.0.1:6805/3169280041] exists,up d3ae3f82-978a-4c7a-8e42-1ef2b1cb17eb 2024-03-09T15:56:41.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:56:41.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:56:41.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:56:41.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:56:41.643 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:56:41.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8af9cd64-80ac-4716-b25a-4b41be89a93e' 2024-03-09T15:56:41.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:56:41.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5huxlaqQfJxAAS/PwYKBvuhVP9IN+2GNLKA== 2024-03-09T15:56:41.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5huxlaqQfJxAAS/PwYKBvuhVP9IN+2GNLKA=="}' 2024-03-09T15:56:41.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8af9cd64-80ac-4716-b25a-4b41be89a93e -i td/osd-scrub-repair/1/new.json 2024-03-09T15:56:41.997 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:56:42.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T15:56:42.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5huxlaqQfJxAAS/PwYKBvuhVP9IN+2GNLKA== --osd-uuid 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:56:42.036 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.037+0000 7f3a345fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:42.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.082+0000 7f3a345fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:42.083 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.083+0000 7f3a345fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:42.083 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.084+0000 7f3a345fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:56:42.083 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.084+0000 7f3a345fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:56:42.083 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.084+0000 7f3a345fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T15:56:42.157 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:42.157+0000 7f3a345fa640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T15:56:47.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T15:56:47.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:56:47.952 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T15:56:47.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T15:56:47.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:56:48.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T15:56:48.305 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T15:56:48.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:56:48.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:56:48.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:56:48.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:56:48.331 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:48.331+0000 7f1703410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:48.333 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:48.333+0000 7f1703410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:48.336 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:48.334+0000 7f1703410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:56:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:48.580 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:56:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:56:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:48.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:48.855 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:48.856+0000 7f1703410640 -1 Falling back to public interface 2024-03-09T15:56:49.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:49.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:49.853 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:56:49.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:56:49.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:49.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:50.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:51.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:56:51.135 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:56:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:51.334 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:56:51.334+0000 7f1703410640 -1 osd.1 0 log_to_monitors true 2024-03-09T15:56:51.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:52.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:52.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:52.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:56:52.453 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:56:52.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:52.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:52.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:53.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:53.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:53.738 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:56:53.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:56:53.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:53.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:54.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:55.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:55.036 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:56:55.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:55.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:56:55.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:55.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:55.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:56.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:56.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:56.319 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:56:56.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:56:56.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:56.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:56.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:57.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:57.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:57.601 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:56:57.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:56:57.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:57.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:57.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:56:58.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:56:58.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:56:58.885 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:56:58.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:56:58.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:56:58.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:56:59.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:00.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:00.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:00.307 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:57:00.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:57:00.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:00.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:00.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:01.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:01.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:01.591 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:57:01.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:57:01.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:01.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:01.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:02.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:02.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:02.875 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:57:02.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:57:02.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:02.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:03.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:04.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:04.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:04.155 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:57:04.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:57:04.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:04.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:04.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:05.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:05.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:57:05.817 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:57:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:05.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:06.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:07.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:07.102 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:57:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:57:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:07.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:07.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:08.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:08.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:08.522 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:57:08.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:57:08.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:08.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:09.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:10.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:10.270 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:57:10.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:10.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:57:10.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:10.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:10.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:11.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:11.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:11.938 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:57:11.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:57:11.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:11.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:12.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:13.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:13.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:13.462 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:57:13.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:57:13.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:13.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:14.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:14.876 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:57:14.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:14.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:57:14.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:14.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:57:15.155 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1346063632,v1:127.0.0.1:6811/1346063632] [v2:127.0.0.1:6812/1346063632,v1:127.0.0.1:6813/1346063632] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:57:15.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:57:15.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:57:15.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:57:15.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T15:57:15.163 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T15:57:15.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cdc983f5-20c2-4f42-a396-8b51eb69c886' 2024-03-09T15:57:15.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:57:15.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbhuxlWeORChAA6zSGLjwraJPzeAZJQCsnNg== 2024-03-09T15:57:15.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbhuxlWeORChAA6zSGLjwraJPzeAZJQCsnNg=="}' 2024-03-09T15:57:15.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cdc983f5-20c2-4f42-a396-8b51eb69c886 -i td/osd-scrub-repair/2/new.json 2024-03-09T15:57:15.527 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:57:15.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T15:57:15.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbhuxlWeORChAA6zSGLjwraJPzeAZJQCsnNg== --osd-uuid cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T15:57:15.567 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.568+0000 7f217a48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:15.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.595+0000 7f217a48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:15.596 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.596+0000 7f217a48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:15.596 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.597+0000 7f217a48b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:57:15.596 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.597+0000 7f217a48b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:57:15.596 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.597+0000 7f217a48b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T15:57:15.662 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:15.662+0000 7f217a48b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T15:57:21.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T15:57:21.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:57:21.333 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T15:57:21.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T15:57:21.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:57:21.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T15:57:21.686 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T15:57:21.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:57:21.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:57:21.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:57:21.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:57:21.711 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:21.712+0000 7f54ac697640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:21.713 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:21.714+0000 7f54ac697640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:21.714 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:21.715+0000 7f54ac697640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:21.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:57:21.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:21.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:57:21.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:21.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:22.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:22.236 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:22.237+0000 7f54ac697640 -1 Falling back to public interface 2024-03-09T15:57:23.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:23.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:23.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:57:23.229 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:57:23.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:23.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:23.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:24.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:24.514 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:57:24.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:24.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:57:24.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:24.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:24.624 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:24.625+0000 7f54ac697640 -1 osd.2 0 log_to_monitors true 2024-03-09T15:57:24.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:25.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:25.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:25.835 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:57:25.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:57:25.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:25.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:26.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:27.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:27.119 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:57:27.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:27.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:57:27.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:27.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:27.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:28.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:28.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:28.403 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:57:28.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:57:28.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:28.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:28.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:29.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:29.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:29.707 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:57:29.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:57:29.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:29.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:30.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:31.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:31.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:31.039 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:57:31.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:57:31.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:31.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:31.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:32.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:32.437 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:57:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:57:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:32.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:33.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:33.719 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:57:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:57:33.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:33.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:34.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:35.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:35.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:35.094 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:57:35.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:57:35.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:35.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:35.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:36.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:36.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:36.376 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:57:36.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:57:36.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:36.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:36.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:37.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:37.941 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:57:37.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:37.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:57:37.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:37.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:38.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:39.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:39.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:39.448 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:57:39.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:57:39.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:39.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:39.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:40.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:40.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:40.848 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:57:40.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:57:40.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:40.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:41.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:42.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:42.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:42.275 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:57:42.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:57:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:43.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:44.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:44.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:44.006 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:57:44.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:57:44.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:44.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:44.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:45.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:45.289 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:57:45.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:45.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:57:45.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:45.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:45.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:46.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:46.647 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T15:57:46.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:46.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T15:57:46.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:46.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:47.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:48.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:48.120 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T15:57:48.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:48.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T15:57:48.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:48.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:57:48.391 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/892906513,v1:127.0.0.1:6819/892906513] [v2:127.0.0.1:6820/892906513,v1:127.0.0.1:6821/892906513] exists,up cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:57:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:57:48.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:57:48.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 701c1fbf-f524-45e7-8951-9d61b269fff9' 2024-03-09T15:57:48.399 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:57:48.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T15:57:48.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8huxlAbqlGBAAKE5kPa/YM0O3G7zd6jYoUg== 2024-03-09T15:57:48.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8huxlAbqlGBAAKE5kPa/YM0O3G7zd6jYoUg=="}' 2024-03-09T15:57:48.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 701c1fbf-f524-45e7-8951-9d61b269fff9 -i td/osd-scrub-repair/3/new.json 2024-03-09T15:57:48.742 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:57:48.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T15:57:48.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8huxlAbqlGBAAKE5kPa/YM0O3G7zd6jYoUg== --osd-uuid 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:57:48.783 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.783+0000 7fd670f73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.819+0000 7fd670f73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.820+0000 7fd670f73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.820+0000 7fd670f73640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T15:57:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.820+0000 7fd670f73640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T15:57:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.820+0000 7fd670f73640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T15:57:48.885 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:48.885+0000 7fd670f73640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T15:57:54.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T15:57:54.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T15:57:54.914 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T15:57:54.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T15:57:54.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T15:57:55.269 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T15:57:55.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T15:57:55.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:57:55.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T15:57:55.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T15:57:55.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T15:57:55.295 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:55.296+0000 7f4765899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:55.297 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:55.298+0000 7f4765899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:55.298 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:55.299+0000 7f4765899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:57:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:55.544 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T15:57:55.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:57:55.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:55.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:57:55.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:55.823 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:55.824+0000 7f4765899640 -1 Falling back to public interface 2024-03-09T15:57:56.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:56.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:56.814 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T15:57:56.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:57:56.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:56.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:57:57.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:58.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:58.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:58.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:57:58.094 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T15:57:58.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:58.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:57:58.156 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:57:58.157+0000 7f4765899640 -1 osd.3 0 log_to_monitors true 2024-03-09T15:57:58.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:57:59.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:57:59.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:57:59.380 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T15:57:59.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:57:59.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:57:59.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:57:59.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:00.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:00.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:00.772 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T15:58:00.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T15:58:00.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:00.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:01.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:02.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:02.184 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T15:58:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T15:58:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:02.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:02.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:03.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:03.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:03.470 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T15:58:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T15:58:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:03.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:04.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:04.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:04.781 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T15:58:04.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T15:58:04.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:04.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:05.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:06.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:06.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:06.215 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T15:58:06.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T15:58:06.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:06.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:06.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:07.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:07.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:07.500 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T15:58:07.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T15:58:07.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:07.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:08.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:08.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:08.834 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T15:58:08.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T15:58:08.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:08.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:09.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:10.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:10.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:10.200 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T15:58:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T15:58:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:10.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:11.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:11.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:11.594 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T15:58:11.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T15:58:11.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:11.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:12.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:13.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:13.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:13.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T15:58:13.481 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T15:58:13.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:13.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:14.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:15.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:15.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:15.342 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T15:58:15.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T15:58:15.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:15.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:17.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:17.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:17.086 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T15:58:17.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T15:58:17.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:17.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:18.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:19.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:19.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:19.099 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T15:58:19.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T15:58:19.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:19.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:19.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:20.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:20.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:20.875 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T15:58:20.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T15:58:20.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:20.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:21.179 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1798050431,v1:127.0.0.1:6827/1798050431] [v2:127.0.0.1:6828/1798050431,v1:127.0.0.1:6829/1798050431] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:58:21.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:58:21.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:58:21.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:58:21.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-09T15:58:21.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T15:58:21.536 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T15:58:21.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T15:58:21.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T15:58:22.021 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T15:58:22.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:58:23.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T15:58:24.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-09T15:58:24.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:58:24.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:58:24.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:58:24.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:58:24.885 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:58:24.885 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:58:24.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:58:24.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:58:24.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:58:24.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:58:25.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:25.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:58:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T15:58:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T15:58:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T15:58:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:25.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:58:25.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T15:58:25.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T15:58:25.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2024-03-09T15:58:25.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:25.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:58:25.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-09T15:58:25.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-09T15:58:25.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509457' 2024-03-09T15:58:25.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:25.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:58:25.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-03-09T15:58:25.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-03-09T15:58:25.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509457 3-85899345926' 2024-03-09T15:58:25.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:25.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T15:58:25.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:25.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:58:25.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T15:58:25.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T15:58:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T15:58:25.744 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836503 2024-03-09T15:58:25.744 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:26.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-09T15:58:26.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:58:27.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:58:27.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:27.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836503 2024-03-09T15:58:27.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:27.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T15:58:27.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:27.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:58:27.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T15:58:27.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:27.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T15:58:27.302 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672980 2024-03-09T15:58:27.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T15:58:27.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:58:27.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-09T15:58:27.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:27.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-09T15:58:27.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:27.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:58:27.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-09T15:58:27.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:27.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-09T15:58:27.620 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509457 2024-03-09T15:58:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-09T15:58:27.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:58:27.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-09T15:58:27.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:27.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-03-09T15:58:27.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:27.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:58:27.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-03-09T15:58:27.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:27.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-03-09T15:58:27.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-03-09T15:58:27.894 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345926 2024-03-09T15:58:27.895 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:58:28.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345926 2024-03-09T15:58:28.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:58:28.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:28.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:58:28.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:58:28.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:58:28.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:58:28.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:58:28.783 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:28.783 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T15:58:29.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-09T15:58:29.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T15:58:29.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T15:58:30.536 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T15:58:30.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T15:58:31.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T15:58:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T15:58:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:58:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:58:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:58:31.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:58:31.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:58:31.551 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:58:31.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:58:31.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:58:31.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:58:31.647 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:31.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:58:32.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-09T15:58:32.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-09T15:58:32.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-09T15:58:32.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:32.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:58:32.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-09T15:58:32.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-09T15:58:32.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672982' 2024-03-09T15:58:32.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:32.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:58:32.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T15:58:32.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T15:58:32.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672982 2-64424509459' 2024-03-09T15:58:32.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:32.332 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:58:32.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345929 2024-03-09T15:58:32.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345929 2024-03-09T15:58:32.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672982 2-64424509459 3-85899345929' 2024-03-09T15:58:32.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:32.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-09T15:58:32.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:32.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:58:32.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-09T15:58:32.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-09T15:58:32.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-09T15:58:32.438 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836506 2024-03-09T15:58:32.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-03-09T15:58:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:58:33.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:58:33.711 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:33.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-09T15:58:33.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:33.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-09T15:58:33.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:33.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:58:33.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-09T15:58:33.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:33.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-09T15:58:33.988 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672982 2024-03-09T15:58:33.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-09T15:58:33.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:58:34.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-09T15:58:34.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:34.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-09T15:58:34.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:34.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:58:34.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-09T15:58:34.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:34.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T15:58:34.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-09T15:58:34.264 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509459 2024-03-09T15:58:34.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:58:34.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-09T15:58:34.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:34.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345929 2024-03-09T15:58:34.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:34.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:58:34.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345929 2024-03-09T15:58:34.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:34.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345929 2024-03-09T15:58:34.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-03-09T15:58:34.538 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345929 2024-03-09T15:58:34.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:58:34.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345929 2024-03-09T15:58:34.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:58:34.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:34.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:58:35.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:58:35.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:58:35.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:58:35.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:35.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:35.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:58:35.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:58:35.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:58:35.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T15:58:35.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T15:58:36.260 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T15:58:36.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T15:58:36.804 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T15:58:36.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T15:58:36.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T15:58:36.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T15:58:37.210 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T15:58:37.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T15:58:37.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T15:58:37.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:58:37.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T15:58:37.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T15:58:37.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T15:58:37.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T15:58:37.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T15:58:37.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T15:58:37.487 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:58:37.487 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T15:58:37.763 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T15:58:37.763 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:37.763 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:58:37.763 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T15:58:37.763 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T15:58:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:58:37.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:58:38.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:58:38.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T15:58:38.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:58:38.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:58:38.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T15:58:38.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:58:38.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:58:38.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T15:58:39.703 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T15:58:40.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T15:58:40.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:58:40.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T15:58:40.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:58:40.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:58:40.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T15:58:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T15:58:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:58:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:58:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:58:40.250 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:40.251+0000 7f8e32a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:40.253 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:40.253+0000 7f8e32a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:40.254 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:40.255+0000 7f8e32a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:40.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:58:40.507 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:58:40.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:40.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:40.766 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:40.767+0000 7f8e32a0d640 -1 Falling back to public interface 2024-03-09T15:58:40.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:41.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:42.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:43.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:43.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:43.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:58:43.066 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:58:43.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:43.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:43.108 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:43.108+0000 7f8e32a0d640 -1 osd.3 37 log_to_monitors true 2024-03-09T15:58:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:44.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:44.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:44.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:58:44.540 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:58:44.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:44.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/1295358702,v1:127.0.0.1:6827/1295358702] [v2:127.0.0.1:6828/1295358702,v1:127.0.0.1:6829/1295358702] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:58:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:58:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:58:45.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:58:45.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:45.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:58:45.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T15:58:45.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T15:58:45.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T15:58:45.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:45.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:58:45.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-09T15:58:45.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-09T15:58:45.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986' 2024-03-09T15:58:45.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:45.734 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:58:45.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-09T15:58:45.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-09T15:58:45.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509463' 2024-03-09T15:58:45.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:58:45.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:58:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-09T15:58:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-09T15:58:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509463 3-176093659138' 2024-03-09T15:58:45.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:45.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T15:58:45.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:45.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:58:45.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T15:58:45.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:45.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T15:58:45.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T15:58:45.945 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836509 2024-03-09T15:58:45.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:46.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T15:58:46.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:58:47.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:58:47.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:58:47.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-03-09T15:58:47.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:47.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-09T15:58:47.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:47.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:58:47.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-09T15:58:47.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:47.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-09T15:58:47.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-09T15:58:47.507 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672986 2024-03-09T15:58:47.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:58:47.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-03-09T15:58:47.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:47.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-09T15:58:47.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:47.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:58:47.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-09T15:58:47.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:47.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-09T15:58:47.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-09T15:58:47.804 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509463 2024-03-09T15:58:47.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:58:48.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-09T15:58:48.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:58:48.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-03-09T15:58:48.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:58:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:58:48.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-03-09T15:58:48.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:58:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-09T15:58:48.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-03-09T15:58:48.081 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 176093659138 2024-03-09T15:58:48.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:58:48.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-09T15:58:48.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:58:48.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:48.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:48.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:58:48.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:58:48.695 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:58:48.695 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:58:48.695 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:58:48.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:58:48.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:58:48.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:58:48.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:58:48.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:58:48.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:58:48.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:58:49.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:58:49.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:58:49.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:58:49.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T15:58:49.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T15:58:49.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:58:49.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:58:49.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:58:49.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:49.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:49.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:49.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T15:58:50.107 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T15:58:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T15:58:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:58:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:50.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:30.311038+0000 '>' 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:50.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:58:51.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:58:51.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:58:51.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:58:51.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:51.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:51.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:51.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:51.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:30.311038+0000 '>' 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:51.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:52.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:52.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:52.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:30.311038+0000 '>' 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:52.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:58:53.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:58:53.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:58:53.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:58:53.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:53.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:53.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:53.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:54.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:30.311038+0000 '>' 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:54.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:58:55.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:58:55.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:58:55.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:58:55.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:58:55.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:58:55.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:58:55.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:58:55.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:50.238257+0000 '>' 2024-03-09T15:58:30.311038+0000 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:58:55.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:58:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:58:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:58:55.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:58:55.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T15:58:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T15:58:57.776 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:58:57.776 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T15:58:57.776 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:58:58.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T15:58:58.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T15:58:58.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:58:58.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T15:58:58.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:58:58.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T15:58:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T15:58:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:58:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:58:58.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:58:58.076 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:58.076+0000 7fbbe02f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:58.078 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:58.079+0000 7fbbe02f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:58.079 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:58.080+0000 7fbbe02f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:58:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T15:58:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:58:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:58:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:58.598 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:58:58.599+0000 7fbbe02f9640 -1 Falling back to public interface 2024-03-09T15:58:58.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:58:59.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:58:59.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:00.822 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:00.822+0000 7fbbe02f9640 -1 osd.3 43 log_to_monitors true 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:00.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 48 up_thru 41 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/4066768541,v1:127.0.0.1:6827/4066768541] [v2:127.0.0.1:6828/4066768541,v1:127.0.0.1:6829/4066768541] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:59:01.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:59:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:59:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:59:01.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:59:01.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:59:01.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:59:01.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:59:01.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:59:01.746 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:59:01.747 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:01.747 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:59:01.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:59:01.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:01.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:59:02.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-09T15:59:02.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-09T15:59:02.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-09T15:59:02.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:02.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:59:02.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-09T15:59:02.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-09T15:59:02.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672990' 2024-03-09T15:59:02.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:02.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:59:02.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-09T15:59:02.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-09T15:59:02.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672990 2-64424509467' 2024-03-09T15:59:02.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:02.392 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:59:02.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-09T15:59:02.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-09T15:59:02.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672990 2-64424509467 3-206158430210' 2024-03-09T15:59:02.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:02.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-09T15:59:02.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:02.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:59:02.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-09T15:59:02.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-09T15:59:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-09T15:59:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836514 2024-03-09T15:59:02.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:02.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836514 2024-03-09T15:59:02.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:59:03.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:59:03.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-09T15:59:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:04.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-09T15:59:04.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:04.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:59:04.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-09T15:59:04.102 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:04.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-09T15:59:04.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-09T15:59:04.104 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672990 2024-03-09T15:59:04.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-03-09T15:59:04.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:04.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-09T15:59:04.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:04.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:59:04.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-09T15:59:04.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-09T15:59:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-09T15:59:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509467 2024-03-09T15:59:04.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:04.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-09T15:59:04.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:04.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-206158430210 2024-03-09T15:59:04.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:04.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:59:04.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-206158430210 2024-03-09T15:59:04.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:04.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-09T15:59:04.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 206158430210' 2024-03-09T15:59:04.654 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 206158430210 2024-03-09T15:59:04.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:59:04.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-09T15:59:04.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:59:04.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:04.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:05.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:59:05.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:05.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:05.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:59:05.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:05.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:05.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:59:05.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:05.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:05.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:06.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T15:59:06.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:06.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:06.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T15:59:06.585 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T15:59:06.586 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T15:59:06.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T15:59:06.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T15:59:06.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T15:59:06.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:59:06.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T15:59:07.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:59:07.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:07.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:07.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:07.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:07.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:07.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:07.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:07.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:59:07.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:07.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:07.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:07.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:59:07.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:59:07.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:59:07.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:59:07.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:59:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-09T15:59:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T15:59:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T15:59:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:07.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:08.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:08.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:08.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:08.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:08.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:59:08.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:08.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:08.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T15:59:09.918 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:10.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:10.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:10.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:59:10.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:59:10.467 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:10.467+0000 7ff2b8b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:10.469 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:10.470+0000 7ff2b8b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:10.470 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:10.471+0000 7ff2b8b19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:10.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:59:10.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:59:10.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:59:10.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:59:10.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:59:10.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:10.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:59:10.723 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:59:10.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:10.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:10.985 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:10.986+0000 7ff2b8b19640 -1 Falling back to public interface 2024-03-09T15:59:10.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:11.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:12.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:12.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:59:12.010 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:59:12.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:12.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:12.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:13.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:13.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:13.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:59:13.276 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:13.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:13.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:13.524 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:13.524+0000 7ff2b8b19640 -1 osd.1 50 log_to_monitors true 2024-03-09T15:59:13.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:14.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/2660220494,v1:127.0.0.1:6811/2660220494] [v2:127.0.0.1:6812/2660220494,v1:127.0.0.1:6813/2660220494] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:59:14.970 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:59:14.971 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:59:14.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:59:14.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:59:14.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:59:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:59:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:59:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:59:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:59:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:59:15.056 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:59:15.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:15.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:59:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-03-09T15:59:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-03-09T15:59:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-03-09T15:59:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:15.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:59:15.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-03-09T15:59:15.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-03-09T15:59:15.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-236223201282' 2024-03-09T15:59:15.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:15.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:59:15.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-09T15:59:15.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-09T15:59:15.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-236223201282 2-64424509471' 2024-03-09T15:59:15.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:15.830 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-236223201282 2-64424509471 3-206158430213' 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-03-09T15:59:15.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:15.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:59:15.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-03-09T15:59:15.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:15.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-03-09T15:59:15.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-03-09T15:59:15.936 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836517 2024-03-09T15:59:15.936 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836517 2024-03-09T15:59:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:59:17.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:59:17.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836517 2024-03-09T15:59:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-03-09T15:59:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:59:17.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-03-09T15:59:17.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:17.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-03-09T15:59:17.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-03-09T15:59:17.497 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 236223201282 2024-03-09T15:59:17.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:17.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-03-09T15:59:17.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:17.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-09T15:59:17.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:59:17.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-09T15:59:17.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:17.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-09T15:59:17.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-09T15:59:17.771 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509471 2024-03-09T15:59:17.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:18.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-03-09T15:59:18.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:18.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-206158430213 2024-03-09T15:59:18.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:18.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:59:18.047 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-206158430213 2024-03-09T15:59:18.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:18.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-03-09T15:59:18.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 206158430213' 2024-03-09T15:59:18.049 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 206158430213 2024-03-09T15:59:18.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:59:18.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430213 2024-03-09T15:59:18.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:59:18.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:18.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:18.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:18.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:59:18.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:18.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:18.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:19.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:59:19.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:59:19.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:59:19.332 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T15:59:19.332 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T15:59:19.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:59:19.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:59:19.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:19.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:19.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:19.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:19.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:58:50.238257+0000 2024-03-09T15:59:19.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T15:59:20.137 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T15:58:50.238257+0000 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:58:50.238257+0000 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:59:20.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:20.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:20.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:20.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:20.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:20.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:20.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:58:50.238257+0000 '>' 2024-03-09T15:58:50.238257+0000 2024-03-09T15:59:20.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:21.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:21.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:21.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:21.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:21.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:21.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:21.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:21.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:58:50.238257+0000 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:21.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:59:21.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T15:59:23.654 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T15:59:23.655 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T15:59:23.655 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T15:59:23.655 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T15:59:23.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:23.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:23.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T15:59:23.925 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:59:23.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:59:23.952 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:23.952+0000 7f9256f21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:23.955 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:23.954+0000 7f9256f21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:23.956 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:23.956+0000 7f9256f21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T15:59:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:59:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:24.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:59:24.473 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:24.473+0000 7f9256f21640 -1 Falling back to public interface 2024-03-09T15:59:24.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:25.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:59:25.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:26.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T15:59:26.639+0000 7f9256f21640 -1 osd.3 57 log_to_monitors true 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:26.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:59:27.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:28.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:28.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:28.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:59:28.042 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T15:59:28.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:28.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T15:59:28.590 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 58 last_clean_interval [48,57) [v2:127.0.0.1:6826/2584039844,v1:127.0.0.1:6827/2584039844] [v2:127.0.0.1:6828/2584039844,v1:127.0.0.1:6829/2584039844] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:59:28.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:59:28.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:59:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:59:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T15:59:28.952 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T15:59:28.952 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T15:59:28.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:59:28.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:28.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:59:29.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-03-09T15:59:29.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-03-09T15:59:29.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-03-09T15:59:29.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:29.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:59:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201286 2024-03-09T15:59:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201286 2024-03-09T15:59:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201286' 2024-03-09T15:59:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:29.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:59:29.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509475 2024-03-09T15:59:29.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509475 2024-03-09T15:59:29.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201286 2-64424509475' 2024-03-09T15:59:29.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:29.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:59:29.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-03-09T15:59:29.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-03-09T15:59:29.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201286 2-64424509475 3-266287972354' 2024-03-09T15:59:29.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:29.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-03-09T15:59:29.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:29.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:59:29.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-03-09T15:59:29.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:29.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-03-09T15:59:29.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-03-09T15:59:29.387 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836521 2024-03-09T15:59:29.388 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:29.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-03-09T15:59:29.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:29.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201286 2024-03-09T15:59:29.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:29.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:59:29.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201286 2024-03-09T15:59:29.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:29.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201286 2024-03-09T15:59:29.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201286' 2024-03-09T15:59:29.661 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 236223201286 2024-03-09T15:59:29.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:29.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201285 -lt 236223201286 2024-03-09T15:59:29.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:59:30.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:59:30.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:31.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201286 -lt 236223201286 2024-03-09T15:59:31.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:31.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509475 2024-03-09T15:59:31.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:31.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:59:31.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509475 2024-03-09T15:59:31.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:31.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509475 2024-03-09T15:59:31.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-03-09T15:59:31.222 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509475 2024-03-09T15:59:31.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:31.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509475 -lt 64424509475 2024-03-09T15:59:31.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:31.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972354 2024-03-09T15:59:31.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:31.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:59:31.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972354 2024-03-09T15:59:31.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:31.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-03-09T15:59:31.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972354' 2024-03-09T15:59:31.500 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 266287972354 2024-03-09T15:59:31.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:59:31.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-03-09T15:59:31.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:59:31.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:31.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:32.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:59:32.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:32.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:32.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:32.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:32.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:32.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:32.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:32.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:59:32.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:32.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:32.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:32.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:59:32.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:59:32.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:59:32.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T15:59:32.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165480"' 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165480' 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T15:59:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165482"' 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165482' 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165480 165482' 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165480 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165483: /' 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T15:59:32.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165485: /' 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:35.620 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:59:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:59:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:59:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: start osd.1 2024-03-09T15:59:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:59:35.759 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.759 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:35.759 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T15:59:35.759 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:35.759 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T15:59:35.760 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:35.761 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:35.762 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:35.763 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:59:35.764 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T15:59:35.764 INFO:tasks.workunit.client.0.smithi146.stderr:165485: start osd.2 2024-03-09T15:59:35.764 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T15:59:40.172 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T15:59:40.173 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T15:59:40.173 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:59:40.173 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:59:40.173 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:59:40.173 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2024-03-09T15:59:35.652+0000 7f650ba12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2024-03-09T15:59:35.655+0000 7f650ba12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2024-03-09T15:59:35.656+0000 7f650ba12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:59:40.174 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 0 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2024-03-09T15:59:36.175+0000 7f650ba12640 -1 Falling back to public interface 2024-03-09T15:59:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 1 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2024-03-09T15:59:38.812+0000 7f650ba12640 -1 osd.1 64 log_to_monitors true 2024-03-09T15:59:40.177 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 3 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.178 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165483: osd.1 up in weight 1 up_from 70 up_thru 58 down_at 65 last_165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2024-03-09T15:59:35.788+0000 7fb187c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2024-03-09T15:59:35.791+0000 7fb187c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2024-03-09T15:59:35.792+0000 7fb187c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T15:59:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 0 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2024-03-09T15:59:36.309+0000 7fb187c68640 -1 Falling back to public interface 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 1 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2024-03-09T15:59:39.175+0000 7fb187c68640 -1 osd.2 64 log_to_monitors true 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 3 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T15:59:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165485: osd.2 up in weight 1 up_from 71 up_thru 51 down_at 65 last_clean_interval [55,64) [v2:127.0.0.1:6810/2866477397,v1:127.0.0.1:6811/2866477397] [v2:127.0.0.1:6812/2866477397,v1:127.0.0.1:6813/2866477397] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:59:41.020 INFO:tasks.workunit.client.0.smithi146.stderr:165483: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 1 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 2 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: 3' 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:59:41.021 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-300647710722' 2024-03-09T15:59:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,64) [v2:127.0.0.1:6818/1344319357,v1:127.0.0.1:6819/1344319357] [v2:127.0.0.1:6820/1344319357,v1:127.0.0.1:6821/1344319357] exists,up cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T15:59:41.130 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 1 2024-03-09T15:59:41.131 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 2 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: 3' 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710723 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710723 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-300647710723' 2024-03-09T15:59:41.132 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-300647710722 2-304942678018' 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972357 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972357 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-300647710722 2-304942678018 3-266287972357' 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:59:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: waiting osd.0 seq 21474836524 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-300647710722 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-300647710722 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-03-09T15:59:42.032 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 300647710722' 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: waiting osd.1 seq 300647710722 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710722 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-304942678018 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:59:42.033 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-304942678018 2024-03-09T15:59:43.537 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-03-09T15:59:43.537 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678019 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678019 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-300647710723 2-304942678019' 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972358 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972358 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-300647710723 2-304942678019 3-266287972358' 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T15:59:43.538 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: waiting osd.0 seq 21474836525 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-300647710723 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-300647710723 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710723 2024-03-09T15:59:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 300647710723' 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: waiting osd.1 seq 300647710723 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710723 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710723 2024-03-09T15:59:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:44.563 INFO:tasks.workunit.client.0.smithi146.stderr:16test/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: waiting osd.2 seq 304942678018 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 304942678018 2024-03-09T15:59:44.564 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678019 -lt 304942678018 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972357 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:44.565 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972357 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972357 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972357' 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: waiting osd.3 seq 266287972357 2024-03-09T15:59:44.566 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972357 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:44.567 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T15:59:44.568 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:44.770 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/q5485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-304942678019 2024-03-09T15:59:44.770 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-304942678019 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678019 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 304942678019' 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: waiting osd.2 seq 304942678019 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678019 -lt 304942678019 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972358 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972358 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972358 2024-03-09T15:59:44.771 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972358' 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: waiting osd.3 seq 266287972358 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972358 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:44.772 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T15:59:44.773 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:45.675 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clonea/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:45.675 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:45.676 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:165483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T15:59:45.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165482 2024-03-09T15:59:45.889 INFO:tasks.workunit.client.0.smithi146.stderr:.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:45.889 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T15:59:45.890 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:165485: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T15:59:45.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T15:59:45.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:46.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:46.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:46.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T15:59:46.693 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T15:59:46.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:46.705 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:46.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:46.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:47.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:47.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:47.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:47.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:47.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:47.972 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:47.972 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:48.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:48.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:49.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:49.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:49.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:49.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:50.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:50.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:50.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:50.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:50.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:50.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:50.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:50.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:50.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:51.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:51.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:52.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:20.640884+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:52.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T15:59:53.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T15:59:53.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T15:59:53.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T15:59:53.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T15:59:53.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T15:59:53.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T15:59:53.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T15:59:53.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:47.294957+0000 '>' 2024-03-09T15:59:20.640884+0000 2024-03-09T15:59:53.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168455"' 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168455' 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168457"' 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168457' 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:59:53.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 168455 168457' 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168455 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168458: /' 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T15:59:53.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168460: /' 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:55.954 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:59:55.955 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: _ 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: hinfo_key 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: snapset 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:55.956 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:55.957 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:55.958 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:55.959 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:55.959 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:55.959 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T15:59:55.959 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T15:59:55.959 INFO:tasks.workunit.client.0.smithi146.stderr:168460: start osd.2 2024-03-09T15:59:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T15:59:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T15:59:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: _ 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: hinfo_key 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: snapset 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T15:59:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T15:59:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T15:59:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T15:59:56.001 INFO:tasks.workunit.client.0.smithi146.stderr:168458: start osd.1 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:00.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2024-03-09T15:59:55.987+0000 7f7575804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2024-03-09T15:59:55.989+0000 7f7575804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2024-03-09T15:59:55.990+0000 7f7575804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 0 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2024-03-09T15:59:56.509+0000 7f7575804640 -1 Falling back to public interface 2024-03-09T16:00:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 1 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2024-03-09T15:59:59.030+0000 7f7575804640 -1 osd.2 74 log_to_monitors true 2024-03-09T16:00:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 3 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.494 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168460: osd.2 up in weight 1 up_from 80 up_thru 51 down_aovery_ops 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2024-03-09T15:59:56.028+0000 7f33ad793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2024-03-09T15:59:56.030+0000 7f33ad793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2024-03-09T15:59:56.032+0000 7f33ad793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:00.495 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 0 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2024-03-09T15:59:56.557+0000 7f33ad793640 -1 Falling back to public interface 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:00.496 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 1 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2024-03-09T15:59:59.165+0000 7f33ad793640 -1 osd.1 74 log_to_monitors true 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 3 2024-03-09T16:00:00.497 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:00.498 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168458: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 75 last_clean_interval [71,74) [v2:127.0.0.1:6810/225800704,v1:127.0.0.1:6811/225800704] [v2:127.0.0.1:6812/225800704,v1:127.0.0.1:6813/225800704] exists,up cdc983f5-20c2-4f42-a396-8b51eb69c886 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:01.246 INFO:tasks.workunit.client.0.smithi146.stderr:168460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 1 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 2 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: 3' 2024-03-09T16:00:01.247 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383683 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383683 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-343597383683' 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: t 75 last_clean_interval [70,74) [v2:127.0.0.1:6818/4054530943,v1:127.0.0.1:6819/4054530943] [v2:127.0.0.1:6820/4054530943,v1:127.0.0.1:6821/4054530943] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:01.248 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:01.249 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 1 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 2 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: 3' 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:01.250 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-09T16:00:01.251 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-09T16:00:01.251 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-343597383682' 2024-03-09T16:00:01.251 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_sta ceph tell osd.2 flush_pg_stats 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383683 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383683 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-343597383683 2-343597383683' 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972364 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972364 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-343597383683 2-343597383683 3-266287972364' 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-03-09T16:00:04.488 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: waiting osd.0 seq 21474836530 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:04.489 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836530 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383683 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383683 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:04.490 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383683 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hets: ceph tell osd.2 flush_pg_stats 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-343597383682 2-343597383682' 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972363 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972363 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-343597383682 2-343597383682 3-266287972363' 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-09T16:00:04.491 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: waiting osd.0 seq 21474836531 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836531 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-09T16:00:05.687 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceplpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: waiting osd.1 seq 343597383683 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383683 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383683 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383683 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383683 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383683' 2024-03-09T16:00:05.688 INFO:tasks.workunit.client.0.smithi146.stderr:168460: waiting osd.2 seq 343597383683 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383683 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972364 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972364 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972364 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972364' 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: waiting osd.3 seq 266287972364 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972364 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:05.689 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:05.690 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //homeh-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168458: waiting osd.1 seq 343597383682 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383682 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:05.763 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: waiting osd.2 seq 343597383682 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383682 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972363 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972363 2024-03-09T16:00:05.764 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972363 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972363' 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: waiting osd.3 seq 266287972363 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972363 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:05.765 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168458: ///ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:168460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:06.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:06.396 INFO:tasks.workunit.client.0.smithi146.stderr:home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:06.396 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:06.396 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:06.396 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:168458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:168458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168457 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:00:06.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:00:06.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:00:06.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-09T16:00:06.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171180"' 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171180' 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:00:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171182"' 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171182' 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171180 171182' 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171180 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171183: /' 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:00:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171185: /' 2024-03-09T16:00:09.248 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.248 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:09.249 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:00:09.250 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:00:09.251 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:00:09.251 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:09.252 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: start osd.1 2024-03-09T16:00:09.253 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.730 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:00:09.731 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:00:09.732 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:09.733 INFO:tasks.workunit.client.0.smithi146.stderr:171183: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:00:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:00:09.735 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:00:09.735 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:00:09.735 INFO:tasks.workunit.client.0.smithi146.stderr:171183: start osd.3 2024-03-09T16:00:09.735 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2024-03-09T16:00:09.281+0000 7ff5159f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2024-03-09T16:00:09.283+0000 7ff5159f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:13.700 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2024-03-09T16:00:09.284+0000 7ff5159f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 0 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2024-03-09T16:00:09.810+0000 7ff5159f9640 -1 Falling back to public interface 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:13.701 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 1 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2024-03-09T16:00:12.292+0000 7ff5159f9640 -1 osd.1 82 log_to_monitors true 2024-03-09T16:00:13.702 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 3 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:14.154 INFO:tasks.workunit.client.0.smithi146.stderr:171185: osd.1 up in weight 1 up_from 87 up_thru 87 down_at 83 last_171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:00:14.180 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:00:14.180 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:14.180 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2024-03-09T16:00:09.759+0000 7ff5d980f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2024-03-09T16:00:09.761+0000 7ff5d980f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2024-03-09T16:00:09.762+0000 7ff5d980f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 0 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:14.181 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2024-03-09T16:00:10.278+0000 7ff5d980f640 -1 Falling back to public interface 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 1 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:14.182 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2024-03-09T16:00:13.231+0000 7ff5d980f640 -1 osd.3 82 log_to_monitors true 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 3 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:14.183 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [80,82) [v2:127.0.0.1:6818/4175933398,v1:127.0.0.1:6819/4175933398] [v2:127.0.0.1:6820/4175933398,v1:127.0.0.1:6821/4175933398] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:171185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 1 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 2 2024-03-09T16:00:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:171185: 3' 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-373662154754' 2024-03-09T16:00:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 4 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:16.256 INFO:tasks.workunit.client.0.smithi146.stderr:171183: osd.3 up in weight 1 up_from 90 up_thru 90 down_at 83 last_clean_interval [62,82) [v2:127.0.0.1:6826/3946019096,v1:127.0.0.1:6827/3946019096] [v2:127.0.0.1:6828/3946019096,v1:127.0.0.1:6829/3946019096] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:16.257 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 1 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 2 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: 3' 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-03-09T16:00:16.258 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-03-09T16:00:16.259 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:16.939 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heltell osd.2 flush_pg_stats 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383686 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383686 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-373662154754 2-343597383686' 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-373662154754 2-343597383686 3-386547056642' 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:16.940 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: waiting osd.0 seq 21474836535 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836535 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154754 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:16.941 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154754 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: waiting osd.1 seq 373662154754 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154754 -lt 373662154754 2024-03-09T16:00:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383686 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383686 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383686 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: waiting osd.2 seq 343597383686 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383688 -lt 343597383686 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:18.134 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-386547056642 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-386547056642 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 386547056642' 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: waiting osd.3 seq 386547056642 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056642 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:18.135 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:18.136 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.cpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154755 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154755 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-373662154755' 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383688 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383688 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-373662154755 2-343597383688' 2024-03-09T16:00:18.148 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056643 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056643 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-373662154755 2-343597383688 3-386547056643' 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: waiting osd.0 seq 21474836536 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:18.149 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836536 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836536 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154755 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154755 2024-03-09T16:00:18.150 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:18.519 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ulient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:18.519 INFO:tasks.workunit.client.0.smithi146.stderr:171185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:18.519 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:18.519 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:18.520 INFO:tasks.workunit.client.0.smithi146.stderr:171185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:18.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154755 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154755' 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: waiting osd.1 seq 373662154755 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154755 -lt 373662154755 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383688 2024-03-09T16:00:19.362 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383688 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383688 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383688' 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: waiting osd.2 seq 343597383688 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383688 -lt 343597383688 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-386547056643 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:19.363 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-386547056643 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056643 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 386547056643' 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: waiting osd.3 seq 386547056643 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056643 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:19.364 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:20.005 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:20.005 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:20.005 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:20.005 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:20.005 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:171183: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:171183: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171182 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:00:20.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:00:20.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:00:20.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:00:20.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:00:20.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:00:20.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:20.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:20.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:20.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:00:20.836 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:20.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:20.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:21.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:47.294957+0000 '>' 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:21.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:00:22.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:00:22.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:00:22.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:00:22.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:22.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:22.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:22.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:22.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:47.294957+0000 '>' 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:22.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:23.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:23.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:23.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:47.294957+0000 '>' 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:23.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:00:24.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:00:24.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:00:24.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:00:24.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:24.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:24.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:24.704 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:24.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T15:59:47.294957+0000 '>' 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:24.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:00:25.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:00:25.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:00:25.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:00:25.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:00:25.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:00:25.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:00:25.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:00:26.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:00:21.260134+0000 '>' 2024-03-09T15:59:47.294957+0000 2024-03-09T16:00:26.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:00:26.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174059"' 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174059' 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174061"' 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174061' 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:00:26.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174062: /' 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 174059 174061' 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174059 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:00:26.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174064: /' 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:00:28.884 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:00:28.885 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: _ 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: hinfo_key 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: snapset 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:00:28.886 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:00:28.887 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:00:28.888 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:00:28.889 INFO:tasks.workunit.client.0.smithi146.stderr:174062: start osd.3 2024-03-09T16:00:29.201 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: _ 2024-03-09T16:00:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:174064: hinfo_key 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: snapset 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:00:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:00:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:00:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:00:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:00:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:00:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:174064: start osd.1 2024-03-09T16:00:33.059 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:00:33.059 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:00:33.059 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:00:33.059 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:33.059 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2024-03-09T16:00:28.916+0000 7f2ef3c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2024-03-09T16:00:28.918+0000 7f2ef3c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2024-03-09T16:00:28.919+0000 7f2ef3c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 0 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:33.060 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2024-03-09T16:00:29.440+0000 7f2ef3c01640 -1 Falling back to public interface 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 1 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:33.061 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2024-03-09T16:00:31.912+0000 7f2ef3c01640 -1 osd.3 92 log_to_monitors true 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 3 2024-03-09T16:00:33.062 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2024-03-09T16:00:29.230+0000 7f176db49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.744 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2024-03-09T16:00:29.239+0000 7f176db49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2024-03-09T16:00:29.240+0000 7f176db49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 0 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2024-03-09T16:00:29.770+0000 7f176db49640 -1 Falling back to public interface 2024-03-09T16:00:33.745 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 1 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2 2024-03-09T16:00:33.746 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2024-03-09T16:00:32.462+0000 7f176db49640 -1 osd.1 92 log_to_monitors true 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 3 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:33.747 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174064: osd.1 up in weight 1 up_from 100 up_thru 87 down_s.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: osd.3 up in weight 1 up_from 97 up_thru 97 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6818/797414606,v1:127.0.0.1:6819/797414606] [v2:127.0.0.1:6820/797414606,v1:127.0.0.1:6821/797414606] exists,up 701c1fbf-f524-45e7-8951-9d61b269fff9 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:34.210 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 1 2024-03-09T16:00:34.211 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 2 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: 3' 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-09T16:00:34.212 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-429496729602' 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $iat 93 last_clean_interval [87,92) [v2:127.0.0.1:6826/1848148823,v1:127.0.0.1:6827/1848148823] [v2:127.0.0.1:6828/1848148823,v1:127.0.0.1:6829/1848148823] exists,up 8af9cd64-80ac-4716-b25a-4b41be89a93e 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:00:34.649 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:00:34.650 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 1 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 2 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: 3' 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:00:34.651 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-03-09T16:00:34.652 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-03-09T16:00:34.652 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-429496729603' 2024-03-09T16:00:34.652 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383694 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383694 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-429496729603 2-343597383694' 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-03-09T16:00:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-429496729603 2-343597383694 3-416611827715' 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: waiting osd.0 seq 21474836542 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836542 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:35.570 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-429496729603 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496729603 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 429496729603' 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: waiting osd.1 seq 429496729603 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729603 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383694 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:35.571 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:35.572 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383694 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ds 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383692 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383692 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-429496729602 2-343597383692' 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-03-09T16:00:36.588 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-429496729602 2-343597383692 3-416611827714' 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: waiting osd.0 seq 21474836541 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:00:36.589 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836541 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-429496729602 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496729602 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 429496729602' 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: waiting osd.1 seq 429496729602 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:00:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729602 2024-03-09T16:00:37.186 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /hubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:37.186 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383694 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383694' 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: waiting osd.2 seq 343597383694 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383694 -lt 343597383694 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-416611827715 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-416611827715 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 416611827715' 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: waiting osd.3 seq 416611827715 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:37.187 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:37.188 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-09T16:00:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383692 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383692 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383692 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383692' 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: waiting osd.2 seq 343597383692 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383694 -lt 343597383692 2024-03-09T16:00:37.948 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-416611827714 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-416611827714 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 416611827714' 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: waiting osd.3 seq 416611827714 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:00:37.949 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:37.950 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:00:38.325 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.cliena/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:38.326 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 2 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:38.327 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:00:38.328 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: gt.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:00:39.395 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:39.396 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:00:39.397 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:00:39.770 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceet_num_pgs: ceph --format json status 2024-03-09T16:00:39.770 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:39.770 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:00:39.770 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:00:39.771 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:174064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:39.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:ph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:00:40.109 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:174062: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:00:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174061 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:00:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:00:40.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:40.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:40.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:40.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:00:40.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:00:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:00:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:00:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:00:40.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:00:40.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:00:40.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:00:40.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:00:40.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:00:40.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:00:40.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:00:40.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:00:40.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:00:40.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:00:40.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:00:40.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:00:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:00:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:00:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:00:40.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:00:40.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:00:40.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:00:40.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:00:40.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:00:40.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:00:40.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:00:40.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:00:40.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:00:40.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:00:40.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:00:40.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:00:40.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:00:40.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:00:40.307 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:00:40.307 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:00:40.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:00:40.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:00:40.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:00:40.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:00:40.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:00:40.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:00:40.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:00:40.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:00:40.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:00:40.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:00:40.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:00:40.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:40.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:00:40.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:00:40.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:00:40.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.911 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:00:40.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:00:40.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:00:40.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:40.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:00:40.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:00:40.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:00:41.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:41.885 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:41.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:41.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:00:41.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:00:41.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:00:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:00:41.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91465ef8-dc61-4ab4-a41c-40adb7d13d8e 2024-03-09T16:00:41.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 91465ef8-dc61-4ab4-a41c-40adb7d13d8e' 2024-03-09T16:00:41.921 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 91465ef8-dc61-4ab4-a41c-40adb7d13d8e 2024-03-09T16:00:41.921 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:00:41.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCph+xlgerLNxAAC+LEAQ5pj7pzYLyTDx/zvQ== 2024-03-09T16:00:41.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCph+xlgerLNxAAC+LEAQ5pj7pzYLyTDx/zvQ=="}' 2024-03-09T16:00:41.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91465ef8-dc61-4ab4-a41c-40adb7d13d8e -i td/osd-scrub-repair/0/new.json 2024-03-09T16:00:42.141 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:00:42.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:00:42.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCph+xlgerLNxAAC+LEAQ5pj7pzYLyTDx/zvQ== --osd-uuid 91465ef8-dc61-4ab4-a41c-40adb7d13d8e 2024-03-09T16:00:42.180 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.181+0000 7fcedfa01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:42.192 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.193+0000 7fcedfa01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:42.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.194+0000 7fcedfa01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:42.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.194+0000 7fcedfa01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:00:42.194 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.194+0000 7fcedfa01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:00:42.194 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.194+0000 7fcedfa01640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:00:42.276 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:42.276+0000 7fcedfa01640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:00:47.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:00:47.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:00:47.806 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:00:47.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:00:47.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:00:48.168 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:00:48.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:00:48.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:00:48.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:00:48.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:00:48.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:00:48.194 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:48.195+0000 7f01bbdbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:48.205 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:48.206+0000 7f01bbdbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:48.206 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:48.207+0000 7f01bbdbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:00:48.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:00:48.467 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:00:48.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:48.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:48.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:48.748 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:48.749+0000 7f01bbdbd640 -1 Falling back to public interface 2024-03-09T16:00:49.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:49.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:49.740 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:00:49.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:00:49.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:49.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:50.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:51.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:51.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:51.020 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:00:51.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:00:51.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:51.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:00:51.064+0000 7f01bbdbd640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:00:51.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:52.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:52.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:52.323 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:00:52.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:00:52.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:52.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:52.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:53.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:53.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:53.606 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:00:53.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:00:53.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:53.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:53.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:54.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:54.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:54.978 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:00:54.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:00:54.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:54.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:55.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:00:56.263 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:00:56.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:56.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:56.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:57.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:57.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:57.633 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:00:57.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:00:57.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:57.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:58.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:00:59.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:00:59.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:00:59.007 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:00:59.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:00:59.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:00:59.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:00:59.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:00.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:00.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:00.293 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:01:00.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:01:00.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:00.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:00.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:01.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:01.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:01.622 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:01:01.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:01:01.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:01.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:02.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:03.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:03.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:03.025 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:01:03.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:01:03.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:03.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:03.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:04.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:04.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:04.419 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:01:04.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:01:04.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:04.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:04.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:05.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:05.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:05.986 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:01:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:01:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:06.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:07.693 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:01:07.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:01:07.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:07.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:08.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:09.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:09.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:09.690 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:01:09.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:01:09.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:09.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:10.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:10.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:01:10.972 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:01:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:12.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:12.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:12.533 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:01:12.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:01:12.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:12.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:13.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:14.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:14.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:14.203 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:01:14.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:01:14.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:14.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:01:14.490 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1305451883,v1:127.0.0.1:6803/1305451883] [v2:127.0.0.1:6804/1305451883,v1:127.0.0.1:6805/1305451883] exists,up 91465ef8-dc61-4ab4-a41c-40adb7d13d8e 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:01:14.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:01:14.495 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:01:14.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1b547d29-7a13-40fc-a5ce-32f7916a5837' 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:01:14.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:01:14.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKh+xlEQSCHhAAf51ExO55nb0fp1dFiKP5Lw== 2024-03-09T16:01:14.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKh+xlEQSCHhAAf51ExO55nb0fp1dFiKP5Lw=="}' 2024-03-09T16:01:14.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b547d29-7a13-40fc-a5ce-32f7916a5837 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:01:14.836 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:01:14.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:01:14.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKh+xlEQSCHhAAf51ExO55nb0fp1dFiKP5Lw== --osd-uuid 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:01:14.876 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.876+0000 7fbd9c20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:14.921 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.921+0000 7fbd9c20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.922+0000 7fbd9c20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.923+0000 7fbd9c20f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:01:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.923+0000 7fbd9c20f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:01:14.922 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.923+0000 7fbd9c20f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:01:14.995 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:14.996+0000 7fbd9c20f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:01:20.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:01:20.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:01:20.851 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:01:20.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:01:20.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:01:21.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:01:21.202 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:01:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:01:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:01:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:01:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:01:21.227 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:21.228+0000 7f0120017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:21.229 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:21.230+0000 7f0120017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:21.230 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:21.231+0000 7f0120017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:21.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:01:21.477 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:01:21.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:21.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:21.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:21.753 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:21.754+0000 7f0120017640 -1 Falling back to public interface 2024-03-09T16:01:22.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:22.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:22.748 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:01:22.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:01:22.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:22.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:23.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:24.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:24.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:24.029 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:01:24.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:01:24.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:24.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:24.200 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:24.200+0000 7f0120017640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:01:24.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:25.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:25.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:25.316 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:01:25.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:01:25.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:25.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:25.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:26.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:26.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:26.598 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:01:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:01:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:26.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:27.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:27.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:27.875 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:01:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:01:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:28.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:29.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:29.207 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:01:29.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:01:29.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:29.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:29.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:30.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:30.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:30.488 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:01:30.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:01:30.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:30.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:30.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:31.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:31.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:31.771 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:01:31.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:01:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:32.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:33.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:33.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:33.189 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:01:33.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:01:33.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:33.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:33.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:34.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:34.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:34.472 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:01:34.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:01:34.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:34.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:34.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:35.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:35.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:35.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:01:35.753 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:01:35.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:35.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:36.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:37.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:37.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:37.169 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:01:37.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:01:37.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:37.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:37.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:38.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:38.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:38.768 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:01:38.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:01:38.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:38.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:39.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:40.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:40.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:40.123 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:01:40.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:01:40.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:40.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:40.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:41.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:41.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:41.884 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:01:41.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:01:41.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:41.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:42.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:43.319 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:01:43.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:01:43.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:43.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:43.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:44.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:44.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:44.971 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:01:44.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:01:44.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:44.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:45.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:46.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:46.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:46.439 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:01:46.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:01:46.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:46.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:46.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:47.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:47.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:47.868 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:01:47.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:01:47.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:47.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:01:48.151 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/133494439,v1:127.0.0.1:6811/133494439] [v2:127.0.0.1:6812/133494439,v1:127.0.0.1:6813/133494439] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:01:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:01:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:01:48.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4f18bb38-464f-4eab-9223-90bd8b3dfd43' 2024-03-09T16:01:48.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:01:48.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsh+xlpLlKChAA7acrV007++49N6qZQfQA4A== 2024-03-09T16:01:48.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsh+xlpLlKChAA7acrV007++49N6qZQfQA4A=="}' 2024-03-09T16:01:48.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f18bb38-464f-4eab-9223-90bd8b3dfd43 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:01:48.492 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:01:48.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:01:48.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsh+xlpLlKChAA7acrV007++49N6qZQfQA4A== --osd-uuid 4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:01:48.532 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.532+0000 7f588f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.585+0000 7f588f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.586+0000 7f588f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.586+0000 7f588f40f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:01:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.586+0000 7f588f40f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:01:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.586+0000 7f588f40f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:01:48.662 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:48.662+0000 7f588f40f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:01:54.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:01:54.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:01:54.806 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:01:54.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:01:54.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:01:55.161 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:01:55.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:01:55.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:01:55.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:01:55.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:01:55.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:01:55.187 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:55.187+0000 7f5faf219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:55.189 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:55.189+0000 7f5faf219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:55.190 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:55.190+0000 7f5faf219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:55.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:01:55.436 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:01:55.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:55.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:01:55.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:55.719 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:55.720+0000 7f5faf219640 -1 Falling back to public interface 2024-03-09T16:01:56.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:56.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:56.705 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:01:56.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:01:56.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:56.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:01:56.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:57.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:57.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:57.984 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:01:57.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:01:57.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:57.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:01:58.164 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:01:58.164+0000 7f5faf219640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:01:58.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:01:59.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:01:59.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:01:59.277 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:01:59.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:01:59.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:01:59.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:01:59.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:00.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:00.562 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:02:00.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:00.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:02:00.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:00.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:00.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:01.847 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:02:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:02:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:02.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:03.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:03.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:03.140 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:02:03.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:02:03.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:03.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:03.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:04.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:04.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:04.422 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:02:04.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:02:04.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:04.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:04.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:05.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:05.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:05.792 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:02:05.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:02:05.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:05.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:06.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:07.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:07.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:07.180 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:02:07.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:02:07.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:07.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:07.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:08.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:08.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:08.461 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:02:08.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:02:08.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:08.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:08.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:09.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:09.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:09.785 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:02:09.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:02:09.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:09.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:10.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:11.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:11.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:11.649 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:02:11.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:02:11.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:11.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:12.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:13.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:13.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:13.599 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:02:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:02:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:14.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:15.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:15.241 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:02:15.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:15.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:02:15.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:15.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:15.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:16.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:16.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:16.599 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:02:16.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:02:16.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:16.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:17.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:18.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:02:18.360 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:02:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:19.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:20.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:20.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:20.121 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:02:20.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:02:20.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:20.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:20.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:21.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:21.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:21.510 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:02:21.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:02:21.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:21.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:02:21.784 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2010877546,v1:127.0.0.1:6819/2010877546] [v2:127.0.0.1:6820/2010877546,v1:127.0.0.1:6821/2010877546] exists,up 4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:02:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:02:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:02:21.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:02:21.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bcaa5f20-4d03-44ec-a37f-b16fc3f98361' 2024-03-09T16:02:21.792 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:02:21.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:02:21.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANiOxlC3sGMBAAyDaHk5sMZpk/PYWi49g8kw== 2024-03-09T16:02:21.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANiOxlC3sGMBAAyDaHk5sMZpk/PYWi49g8kw=="}' 2024-03-09T16:02:21.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcaa5f20-4d03-44ec-a37f-b16fc3f98361 -i td/osd-scrub-repair/3/new.json 2024-03-09T16:02:22.224 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:02:22.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T16:02:22.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANiOxlC3sGMBAAyDaHk5sMZpk/PYWi49g8kw== --osd-uuid bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:02:22.266 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.266+0000 7feed7c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:22.317 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.318+0000 7feed7c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:22.318 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.319+0000 7feed7c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:22.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.319+0000 7feed7c87640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:02:22.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.319+0000 7feed7c87640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:02:22.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.319+0000 7feed7c87640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:02:22.385 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:22.385+0000 7feed7c87640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T16:02:28.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T16:02:28.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:02:28.338 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T16:02:28.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T16:02:28.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:02:28.690 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T16:02:28.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T16:02:28.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:02:28.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:02:28.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:02:28.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:02:28.716 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:28.717+0000 7f5e06a87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:28.718 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:28.719+0000 7f5e06a87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:28.719 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:28.720+0000 7f5e06a87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:02:28.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T16:02:28.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:02:28.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:02:28.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:02:28.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:02:28.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:28.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:02:28.968 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:02:28.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:28.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:29.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:29.237 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:29.238+0000 7f5e06a87640 -1 Falling back to public interface 2024-03-09T16:02:30.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:30.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:30.238 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:02:30.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:02:30.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:30.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:30.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:31.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:31.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:31.521 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:02:31.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:02:31.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:31.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:31.580 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:02:31.581+0000 7f5e06a87640 -1 osd.3 0 log_to_monitors true 2024-03-09T16:02:31.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:32.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:32.804 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:02:32.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:32.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:02:32.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:32.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:33.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:34.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:34.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:34.092 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:02:34.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:02:34.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:34.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:34.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:35.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:35.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:35.376 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:02:35.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:02:35.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:35.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:35.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:36.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:36.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:36.658 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:02:36.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:02:36.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:36.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:36.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:37.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:37.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:37.939 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:02:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:02:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:38.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:39.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:39.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:39.242 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:02:39.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:02:39.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:39.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:39.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:40.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:40.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:40.546 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:02:40.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:02:40.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:40.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:40.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:41.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:41.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:41.829 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:02:41.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:02:41.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:41.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:42.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:43.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:43.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:43.149 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:02:43.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:02:43.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:43.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:43.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:44.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:44.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:44.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:02:44.747 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:02:44.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:44.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:45.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:46.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:46.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:46.031 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:02:46.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:02:46.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:46.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:46.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:47.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:47.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:47.336 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:02:47.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:02:47.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:47.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:47.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:48.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:48.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:48.910 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:02:48.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:02:48.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:48.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:49.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:50.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:50.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:50.513 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:02:50.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:02:50.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:50.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:50.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:51.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:51.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:51.854 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:02:51.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:02:51.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:51.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:52.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:53.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:53.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:53.215 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:02:53.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:02:53.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:53.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:53.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:02:54.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:02:54.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:02:54.525 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:02:54.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:02:54.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:02:54.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:02:54.796 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3284481983,v1:127.0.0.1:6827/3284481983] [v2:127.0.0.1:6828/3284481983,v1:127.0.0.1:6829/3284481983] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:02:54.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:02:54.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:02:54.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:02:54.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-09T16:02:54.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:02:55.060 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:02:55.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:02:55.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:02:55.467 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:02:55.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:02:56.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:02:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-09T16:02:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:02:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:02:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:02:58.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:02:58.732 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:02:58.732 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:02:58.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:02:58.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:02:58.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:02:58.827 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:02:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:02:59.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:02:59.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T16:02:59.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T16:02:59.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T16:02:59.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:02:59.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:02:59.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T16:02:59.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T16:02:59.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672978' 2024-03-09T16:02:59.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:02:59.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:02:59.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542159 2024-03-09T16:02:59.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542159 2024-03-09T16:02:59.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672978 2-60129542159' 2024-03-09T16:02:59.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:02:59.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378631 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378631 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672978 2-60129542159 3-81604378631' 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T16:02:59.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:02:59.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:02:59.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T16:02:59.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:02:59.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T16:02:59.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T16:02:59.531 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836509 2024-03-09T16:02:59.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:02:59.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T16:02:59.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:03:00.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:03:00.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:01.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-03-09T16:03:01.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:01.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T16:03:01.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:01.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:03:01.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T16:03:01.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:01.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T16:03:01.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T16:03:01.088 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672978 2024-03-09T16:03:01.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:03:01.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-09T16:03:01.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:01.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542159 2024-03-09T16:03:01.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:01.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:03:01.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542159 2024-03-09T16:03:01.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:01.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542159 2024-03-09T16:03:01.361 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542159 2024-03-09T16:03:01.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2024-03-09T16:03:01.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:03:01.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542159 -lt 60129542159 2024-03-09T16:03:01.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:01.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378631 2024-03-09T16:03:01.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:01.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:03:01.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378631 2024-03-09T16:03:01.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:01.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378631 2024-03-09T16:03:01.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2024-03-09T16:03:01.640 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 81604378631 2024-03-09T16:03:01.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:03:01.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378631 -lt 81604378631 2024-03-09T16:03:01.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:03:01.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:01.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:03:02.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:03:02.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:03:02.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:03:02.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:02.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:02.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:03:02.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:03:02.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:03:02.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-03-09T16:03:02.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:03:02.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:03:02.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T16:03:02.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:03:02.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-09T16:03:03.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:03:03.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:03:04.233 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:03:04.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:03:05.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T16:03:05.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T16:03:05.726 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T16:03:05.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:03:05.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:03:05.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:03:05.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:03:05.739 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:03:05.739 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:03:05.740 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:03:05.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:03:05.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:03:05.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:03:05.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:03:05.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:03:05.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:03:05.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:03:05.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:03:05.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:03:06.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:03:06.327 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:06.327 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:06.327 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:03:06.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:03:06.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:06.328 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:03:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-09T16:03:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-09T16:03:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-09T16:03:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:06.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:03:06.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T16:03:06.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T16:03:06.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672980' 2024-03-09T16:03:06.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:06.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:03:06.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542162 2024-03-09T16:03:06.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542162 2024-03-09T16:03:06.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672980 2-60129542162' 2024-03-09T16:03:06.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:06.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:03:06.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378633 2024-03-09T16:03:06.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378633 2024-03-09T16:03:06.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672980 2-60129542162 3-81604378633' 2024-03-09T16:03:06.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:06.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-09T16:03:06.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:06.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:03:06.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-09T16:03:06.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:06.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-09T16:03:06.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-09T16:03:06.783 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836512 2024-03-09T16:03:06.783 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:07.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836512 2024-03-09T16:03:07.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:03:08.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:03:08.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:08.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-09T16:03:08.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:08.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T16:03:08.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:08.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:03:08.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T16:03:08.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:08.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T16:03:08.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T16:03:08.338 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672980 2024-03-09T16:03:08.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:03:08.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-03-09T16:03:08.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:08.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542162 2024-03-09T16:03:08.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:08.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:03:08.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542162 2024-03-09T16:03:08.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:08.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542162 2024-03-09T16:03:08.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2024-03-09T16:03:08.612 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542162 2024-03-09T16:03:08.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:03:08.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542162 -lt 60129542162 2024-03-09T16:03:08.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:08.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378633 2024-03-09T16:03:08.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:08.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:03:08.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378633 2024-03-09T16:03:08.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:08.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378633 2024-03-09T16:03:08.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2024-03-09T16:03:08.890 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 81604378633 2024-03-09T16:03:08.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:03:09.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378633 -lt 81604378633 2024-03-09T16:03:09.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:03:09.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:09.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:09.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:03:09.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:03:09.501 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:03:09.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:03:09.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:03:09.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:03:09.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:03:09.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:03:09.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:03:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:03:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:03:10.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T16:03:10.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:03:10.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:03:10.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:03:10.472 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:03:10.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:03:10.858 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:03:10.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:03:10.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:03:10.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T16:03:11.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T16:03:11.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T16:03:11.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T16:03:11.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:03:11.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:03:11.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T16:03:11.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T16:03:11.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:03:11.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T16:03:11.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T16:03:11.409 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:03:11.409 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:03:11.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T16:03:11.684 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:11.684 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:11.684 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:03:11.684 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T16:03:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:03:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:03:11.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:03:11.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:11.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:03:13.658 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:03:14.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:03:14.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:03:14.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:03:14.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:03:14.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:03:14.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:03:14.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:03:14.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:03:14.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:03:14.205 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:14.206+0000 7f52ea2f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:14.208 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:14.208+0000 7f52ea2f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:14.209 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:14.210+0000 7f52ea2f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:14.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:14.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:14.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:14.737 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:14.738+0000 7f52ea2f5640 -1 Falling back to public interface 2024-03-09T16:03:15.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:15.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:15.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:03:15.728 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:15.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:15.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:16.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:17.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:17.079 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:17.080+0000 7f52ea2f5640 -1 osd.3 38 log_to_monitors true 2024-03-09T16:03:17.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:18.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [19,38) [v2:127.0.0.1:6826/3859638430,v1:127.0.0.1:6827/3859638430] [v2:127.0.0.1:6828/3859638430,v1:127.0.0.1:6829/3859638430] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:03:18.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:03:18.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:03:18.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:03:19.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:03:19.113 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:19.114 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:19.114 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:03:19.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:03:19.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:19.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:03:19.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-03-09T16:03:19.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-03-09T16:03:19.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-03-09T16:03:19.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:19.229 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:03:19.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-09T16:03:19.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-09T16:03:19.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672984' 2024-03-09T16:03:19.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:19.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:03:19.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542165 2024-03-09T16:03:19.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542165 2024-03-09T16:03:19.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672984 2-60129542165' 2024-03-09T16:03:19.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:19.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:03:19.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-09T16:03:19.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-09T16:03:19.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-42949672984 2-60129542165 3-184683593730' 2024-03-09T16:03:19.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:19.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-03-09T16:03:19.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:19.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:03:19.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-03-09T16:03:19.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:19.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-03-09T16:03:19.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-03-09T16:03:19.543 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836515 2024-03-09T16:03:19.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:19.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-03-09T16:03:19.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:03:20.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:03:20.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:21.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836515 2024-03-09T16:03:21.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:21.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-09T16:03:21.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:21.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:03:21.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-09T16:03:21.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:21.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-09T16:03:21.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-09T16:03:21.099 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672984 2024-03-09T16:03:21.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:03:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-09T16:03:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:21.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542165 2024-03-09T16:03:21.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:03:21.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542165 2024-03-09T16:03:21.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:21.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542165 2024-03-09T16:03:21.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2024-03-09T16:03:21.374 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542165 2024-03-09T16:03:21.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:03:21.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542165 -lt 60129542165 2024-03-09T16:03:21.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:21.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-184683593730 2024-03-09T16:03:21.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:21.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:03:21.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-184683593730 2024-03-09T16:03:21.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-09T16:03:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-03-09T16:03:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 184683593730 2024-03-09T16:03:21.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:03:21.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-09T16:03:21.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:03:21.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:21.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:03:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:03:22.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:03:22.533 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:03:22.533 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:22.533 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:03:22.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:03:22.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:03:23.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:03:23.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:03:23.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:03:23.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:03:23.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:23.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:23.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:23.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:23.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:23.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:03:23.679 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:23.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:23.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:04.150404+0000 '>' 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:23.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:24.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:24.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:24.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:24.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:24.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:24.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:24.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:25.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:04.150404+0000 '>' 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:25.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:26.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:26.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:26.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:26.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:26.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:26.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:26.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:26.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:04.150404+0000 '>' 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:26.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:27.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:27.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:27.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:27.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:27.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:27.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:27.553 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:04.150404+0000 '>' 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:28.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:28.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:28.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:28.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:28.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:28.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:28.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:24.006113+0000 '>' 2024-03-09T16:03:04.150404+0000 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:03:29.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:03:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:29.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:03:30.863 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:03:30.864 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:03:30.864 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:03:30.864 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:03:31.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:03:31.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:03:31.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:03:31.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:03:31.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:03:31.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:03:31.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:03:31.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:03:31.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:03:31.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:03:31.165 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:31.166+0000 7fef42706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:31.167 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:31.168+0000 7fef42706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:31.169 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:31.169+0000 7fef42706640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:31.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:31.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:31.685 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:31.686+0000 7fef42706640 -1 Falling back to public interface 2024-03-09T16:03:31.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:32.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:32.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:32.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:03:32.695 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:32.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:32.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:32.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:33.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:33.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:33.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:03:33.979 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:33.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:33.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:33.985 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:33.985+0000 7fef42706640 -1 osd.3 45 log_to_monitors true 2024-03-09T16:03:34.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:35.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:03:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:03:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:35.687 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/255048207,v1:127.0.0.1:6827/255048207] [v2:127.0.0.1:6828/255048207,v1:127.0.0.1:6829/255048207] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:03:35.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:03:35.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:03:35.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:03:35.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:03:35.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:03:35.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:03:35.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:03:35.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:03:36.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:36.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:03:36.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-03-09T16:03:36.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-03-09T16:03:36.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-03-09T16:03:36.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:36.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:03:36.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-03-09T16:03:36.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-03-09T16:03:36.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672988' 2024-03-09T16:03:36.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:36.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:03:36.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542170 2024-03-09T16:03:36.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542170 2024-03-09T16:03:36.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672988 2-60129542170' 2024-03-09T16:03:36.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:36.602 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672988 2-60129542170 3-210453397506' 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-03-09T16:03:36.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:36.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:03:36.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-03-09T16:03:36.708 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:36.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-03-09T16:03:36.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-03-09T16:03:36.709 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836520 2024-03-09T16:03:36.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:36.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-03-09T16:03:36.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:03:37.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:03:37.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:38.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-03-09T16:03:38.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:38.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-03-09T16:03:38.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:38.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:03:38.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-03-09T16:03:38.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-03-09T16:03:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-03-09T16:03:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672988 2024-03-09T16:03:38.291 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:03:38.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672988 2024-03-09T16:03:38.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:38.562 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542170 2024-03-09T16:03:38.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:38.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:03:38.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542170 2024-03-09T16:03:38.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542170 2024-03-09T16:03:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542170' 2024-03-09T16:03:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542170 2024-03-09T16:03:38.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:03:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542170 -lt 60129542170 2024-03-09T16:03:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:38.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397506 2024-03-09T16:03:38.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:38.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:03:38.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397506 2024-03-09T16:03:38.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:38.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-09T16:03:38.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397506' 2024-03-09T16:03:38.838 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 210453397506 2024-03-09T16:03:38.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:03:39.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-03-09T16:03:39.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:03:39.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:39.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:39.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:03:39.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:03:39.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:03:39.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:03:39.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:03:39.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:03:39.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:03:39.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:03:39.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:03:39.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:03:39.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:39.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:40.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:03:40.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:03:40.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:03:40.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:03:40.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:03:40.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:03:40.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:03:40.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:03:40.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:03:40.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:03:40.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:03:40.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:03:40.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:03:40.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:03:40.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:03:40.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:03:40.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:03:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:03:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:03:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:03:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:03:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:03:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:03:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:03:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T16:03:42.161 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:03:42.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:03:42.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:03:42.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:03:42.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:03:42.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:03:42.690 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:03:42.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:03:42.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:03:42.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:03:42.721 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:42.720+0000 7fa7b5123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:42.722 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:42.722+0000 7fa7b5123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:42.723 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:42.724+0000 7fa7b5123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:42.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:03:42.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:03:42.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:03:42.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:03:42.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:03:42.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:42.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:03:42.975 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:03:42.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:42.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:03:43.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:43.245 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:43.246+0000 7fa7b5123640 -1 Falling back to public interface 2024-03-09T16:03:44.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:44.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:44.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:03:44.246 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:44.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:44.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:03:44.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:45.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:45.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:45.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:03:45.529 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:45.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:45.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:03:45.532 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:45.533+0000 7fa7b5123640 -1 osd.1 50 log_to_monitors true 2024-03-09T16:03:45.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:03:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:03:46.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:46.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:03:47.105 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/3715420448,v1:127.0.0.1:6811/3715420448] [v2:127.0.0.1:6812/3715420448,v1:127.0.0.1:6813/3715420448] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:03:47.106 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:03:47.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:03:47.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:03:47.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:03:47.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:03:47.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:03:47.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:03:47.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:47.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:03:47.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-03-09T16:03:47.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-03-09T16:03:47.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-03-09T16:03:47.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:47.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:03:47.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-03-09T16:03:47.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-03-09T16:03:47.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-236223201282' 2024-03-09T16:03:47.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:47.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:03:47.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542173 2024-03-09T16:03:47.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542173 2024-03-09T16:03:47.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-236223201282 2-60129542173' 2024-03-09T16:03:47.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:03:47.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:03:47.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-03-09T16:03:47.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-03-09T16:03:47.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-236223201282 2-60129542173 3-210453397509' 2024-03-09T16:03:47.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:47.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-03-09T16:03:47.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:47.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:03:47.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-03-09T16:03:47.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:47.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-03-09T16:03:47.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-03-09T16:03:47.875 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836523 2024-03-09T16:03:47.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:03:48.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-03-09T16:03:48.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:48.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-03-09T16:03:48.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:48.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:03:48.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-03-09T16:03:48.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:48.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-03-09T16:03:48.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-03-09T16:03:48.153 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 236223201282 2024-03-09T16:03:48.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:03:48.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-03-09T16:03:48.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:48.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542173 2024-03-09T16:03:48.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:48.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:03:48.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542173 2024-03-09T16:03:48.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:48.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542173 2024-03-09T16:03:48.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542173' 2024-03-09T16:03:48.436 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542173 2024-03-09T16:03:48.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:03:48.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542173 -lt 60129542173 2024-03-09T16:03:48.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:03:48.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397509 2024-03-09T16:03:48.709 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:03:48.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:03:48.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397509 2024-03-09T16:03:48.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:03:48.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-03-09T16:03:48.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397509' 2024-03-09T16:03:48.712 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 210453397509 2024-03-09T16:03:48.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:03:48.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397509 -lt 210453397509 2024-03-09T16:03:48.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:03:48.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:48.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:03:49.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:03:49.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:03:49.597 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:03:49.597 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:03:49.597 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:03:49.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:03:49.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:03:49.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:50.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:50.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:50.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:03:50.753 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:50.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:51.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:24.006113+0000 '>' 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:51.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:52.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:52.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:52.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:52.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:52.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:52.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:24.006113+0000 '>' 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:52.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:53.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:53.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:53.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:53.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:53.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:53.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:53.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:53.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:24.006113+0000 '>' 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:53.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:54.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:54.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:54.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:54.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:54.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:54.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:54.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:54.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:24.006113+0000 '>' 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:54.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:03:55.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:03:55.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:03:55.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:03:55.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:03:55.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:03:55.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:03:55.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:03:56.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:24.006113+0000 2024-03-09T16:03:56.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:03:56.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:03:56.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:03:56.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:03:56.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:56.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:03:57.909 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:03:57.910 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:03:57.910 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:03:57.910 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:03:58.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:03:58.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:03:58.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:03:58.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:03:58.182 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:03:58.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:03:58.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:03:58.184 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T16:03:58.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:03:58.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:03:58.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:03:58.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:03:58.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:03:58.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:03:58.210 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:58.210+0000 7f321f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:58.212 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:58.212+0000 7f321f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:58.213+0000 7f321f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:03:58.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:58.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:03:58.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:03:58.733 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:03:58.733+0000 7f321f9bb640 -1 Falling back to public interface 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:03:59.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:00.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:00.917 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:04:00.917+0000 7f321f9bb640 -1 osd.3 56 log_to_monitors true 2024-03-09T16:04:01.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:01.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:01.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:01.014 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:04:01.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:01.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:01.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:02.784 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [49,56) [v2:127.0.0.1:6826/3274985841,v1:127.0.0.1:6827/3274985841] [v2:127.0.0.1:6828/3274985841,v1:127.0.0.1:6829/3274985841] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:04:02.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:02.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:02.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:02.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:02.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:02.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:02.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:02.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:02.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:02.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:3' 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:03.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:03.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-03-09T16:04:03.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-03-09T16:04:03.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-03-09T16:04:03.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:03.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:03.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201286 2024-03-09T16:04:03.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201286 2024-03-09T16:04:03.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-236223201286' 2024-03-09T16:04:03.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:03.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:04:03.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542177 2024-03-09T16:04:03.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542177 2024-03-09T16:04:03.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-236223201286 2-60129542177' 2024-03-09T16:04:03.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:03.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-236223201286 2-60129542177 3-257698037762' 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-03-09T16:04:03.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:03.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:03.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-03-09T16:04:03.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:03.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-03-09T16:04:03.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-03-09T16:04:03.571 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836527 2024-03-09T16:04:03.571 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:03.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-03-09T16:04:03.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:04.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:04.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:05.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-03-09T16:04:05.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:05.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201286 2024-03-09T16:04:05.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:05.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:05.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201286 2024-03-09T16:04:05.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:05.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201286 2024-03-09T16:04:05.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201286' 2024-03-09T16:04:05.128 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 236223201286 2024-03-09T16:04:05.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:05.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201286 -lt 236223201286 2024-03-09T16:04:05.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:05.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542177 2024-03-09T16:04:05.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:05.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:05.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542177 2024-03-09T16:04:05.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:05.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542177 2024-03-09T16:04:05.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2024-03-09T16:04:05.404 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542177 2024-03-09T16:04:05.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:05.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542177 -lt 60129542177 2024-03-09T16:04:05.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:05.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-03-09T16:04:05.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:05.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:05.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-03-09T16:04:05.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:05.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-03-09T16:04:05.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-03-09T16:04:05.678 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 257698037762 2024-03-09T16:04:05.678 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:05.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-03-09T16:04:05.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:05.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:05.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:06.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:06.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:06.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:06.310 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:06.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:06.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:06.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:06.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:06.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:06.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:06.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:06.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:06.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:06.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:06.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:06.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:04:06.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:06.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191032"' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191032' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191034"' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191034' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191032 191034' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191032 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191035: /' 2024-03-09T16:04:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T16:04:06.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191037: /' 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:04:09.870 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T16:04:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:191035: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:09.872 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:09.873 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: start osd.1 2024-03-09T16:04:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:04:09.967 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.967 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:09.967 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:09.968 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:09.969 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:09.970 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:09.971 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:04:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:04:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:191037: start osd.2 2024-03-09T16:04:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2024-03-09T16:04:09.998+0000 7fbec53dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2024-03-09T16:04:10.000+0000 7fbec53dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2024-03-09T16:04:10.001+0000 7fbec53dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:14.527 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 0 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2024-03-09T16:04:10.519+0000 7fbec53dd640 -1 Falling back to public interface 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 1 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2024-03-09T16:04:13.389+0000 7fbec53dd640 -1 osd.2 62 log_to_monitors true 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 3 2024-03-09T16:04:14.529 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191037: osd.2 up in weight 1 up_from 68 up_thru 51 down_at 63 last_191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2024-03-09T16:04:09.903+0000 7f9b577fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2024-03-09T16:04:09.905+0000 7f9b577fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2024-03-09T16:04:09.906+0000 7f9b577fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:14.530 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 0 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2024-03-09T16:04:10.425+0000 7f9b577fa640 -1 Falling back to public interface 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 1 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2024-03-09T16:04:13.016+0000 7f9b577fa640 -1 osd.1 62 log_to_monitors true 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:14.533 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 3 2024-03-09T16:04:14.533 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:14.533 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:191035: osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [55,62) [v2:127.0.0.1:6810/4143182283,v1:127.0.0.1:6811/4143182283] [v2:127.0.0.1:6812/4143182283,v1:127.0.0.1:6813/4143182283] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:04:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:15.313 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:15.314 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 1 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 2 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: 3' 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-03-09T16:04:15.315 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-287762808835' 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [14,62) [v2:127.0.0.1:6818/2474539391,v1:127.0.0.1:6819/2474539391] [v2:127.0.0.1:6820/2474539391,v1:127.0.0.1:6821/2474539391] exists,up 4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:15.316 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 1 2024-03-09T16:04:15.317 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 2 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: 3' 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-09T16:04:15.318 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-287762808834' 2024-03-09T16:04:15.319 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-287762808835 2-292057776130' 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-03-09T16:04:17.500 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-287762808835 2-292057776130 3-257698037766' 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: waiting osd.0 seq 21474836532 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836532 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:17.501 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808835 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808835 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: waiting osd.1 seq 287762808835 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-03-09T16:04:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:1910tell osd.2 flush_pg_stats 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776131 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776131 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-287762808834 2-292057776131' 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-03-09T16:04:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-287762808834 2-292057776131 3-257698037765' 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: waiting osd.0 seq 21474836531 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:17.504 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836531 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: waiting osd.1 seq 287762808834 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-03-09T16:04:17.505 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:18.707 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-292057776131 2024-03-09T16:04:18.707 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-292057776131 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776131 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: waiting osd.2 seq 292057776131 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776131 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037765 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037765 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037765' 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: waiting osd.3 seq 257698037765 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037765 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:18.710 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:18.710 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:04:18.710 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:18.790 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.c35: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-292057776130 2024-03-09T16:04:18.790 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-292057776130 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: waiting osd.2 seq 292057776130 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776130 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-03-09T16:04:18.791 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: waiting osd.3 seq 257698037766 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037766 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:18.792 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:18.793 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:18.793 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:04:18.793 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:20.211 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:04:20.212 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:04:20.213 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:20.243 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:20.244 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:04:20.245 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:04:21.053 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:04:21.053 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:21.054 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:21.055 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:21.055 INFO:tasks.workunit.client.0.smithi146.stderr:191037: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:21.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:21.101 INFO:tasks.workunit.client.0.smithi146.stderr:clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:21.102 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:191035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191034 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:04:21.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:04:21.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:04:21.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:21.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:21.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:21.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:04:21.902 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:04:21.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:21.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:04:21.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:21.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:21.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:22.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:22.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:23.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:23.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:23.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:23.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:23.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:23.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:23.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:23.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:23.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:24.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:24.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:24.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:25.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:25.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:25.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:25.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:25.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:25.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:25.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:26.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:26.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:27.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:27.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:27.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:27.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:27.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:27.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:27.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:27.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:03:50.953731+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:27.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:28.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:28.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:28.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:28.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:28.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:28.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:28.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:28.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:04:21.932442+0000 '>' 2024-03-09T16:03:50.953731+0000 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194153"' 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194153' 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194155"' 2024-03-09T16:04:28.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194155' 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194153 194155' 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194156: /' 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194153 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T16:04:28.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194158: /' 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:04:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: _ 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: hinfo_key 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: snapset 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:31.082 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:31.083 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:04:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:04:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:194158: start osd.2 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:31.126 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: _ 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: hinfo_key 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: snapset 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:194156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:04:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:04:31.130 INFO:tasks.workunit.client.0.smithi146.stderr:194156: start osd.1 2024-03-09T16:04:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:04:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:04:35.621 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2024-03-09T16:04:31.112+0000 7f0d15d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2024-03-09T16:04:31.114+0000 7f0d15d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2024-03-09T16:04:31.115+0000 7f0d15d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:04:35.622 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:04:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.623 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 0 2024-03-09T16:04:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:35.624 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2024-03-09T16:04:31.637+0000 7f0d15d24640 -1 Falling back to public interface 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 1 2024-03-09T16:04:35.625 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2024-03-09T16:04:34.321+0000 7f0d15d24640 -1 osd.2 70 log_to_monitors true 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.626 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.627 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:35.627 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 3 2024-03-09T16:04:35.627 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.627 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:04:35.848 INFO:tasks.workunit.client.0.smithi146.stderr:194158: osd.2 up in weight 1 up_from 76 up_thru 51 down_aovery_ops 2024-03-09T16:04:35.848 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:04:35.848 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:04:35.848 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:35.848 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2024-03-09T16:04:31.156+0000 7f906df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2024-03-09T16:04:31.158+0000 7f906df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2024-03-09T16:04:31.159+0000 7f906df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:04:35.849 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:35.850 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 0 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2024-03-09T16:04:31.677+0000 7f906df47640 -1 Falling back to public interface 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.851 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 1 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.852 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2024-03-09T16:04:34.484+0000 7f906df47640 -1 osd.1 70 log_to_monitors true 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 3 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:35.854 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:36.597 INFO:tasks.workunit.client.0.smithi146.stderr:194156: osd.1 up in weight 1 up_from 77 up_thru 68 down_at 71 last_clean_interval [68,70) [v2:127.0.0.1:6810/3268776377,v1:127.0.0.1:6811/3268776377] [v2:127.0.0.1:6812/3268776377,v1:127.0.0.1:6813/3268776377] exists,up 4f18bb38-464f-4eab-9223-90bd8b3dfd43 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:36.598 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 1 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 2 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: 3' 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-09T16:04:36.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-330712481794' 2024-03-09T16:04:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 71 last_clean_interval [67,70) [v2:127.0.0.1:6818/1091126895,v1:127.0.0.1:6819/1091126895] [v2:127.0.0.1:6820/1091126895,v1:127.0.0.1:6821/1091126895] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:36.806 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 1 2024-03-09T16:04:36.807 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 2 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: 3' 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-330712481795' 2024-03-09T16:04:36.808 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-330712481794 2-326417514498' 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-330712481794 2-326417514498 3-257698037772' 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: waiting osd.0 seq 21474836537 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:39.072 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: waiting osd.1 seq 330712481794 2024-03-09T16:04:39.073 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:39.074 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-09T16:04:39.355 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-03-09T16:04:39.355 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-03-09T16:04:39.355 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-330712481795 2-326417514499' 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037773 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037773 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-330712481795 2-326417514499 3-257698037773' 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-03-09T16:04:39.356 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: waiting osd.0 seq 21474836538 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836538 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481795 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481795 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-03-09T16:04:39.357 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481795' 2024-03-09T16:04:39.358 INFO:tasks.workunit.client.0.smithi146.stderr:194156: waiting osd.1 seq 330712481795 2024-03-09T16:04:39.358 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:39.358 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-03-09T16:04:40.373 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:40.373 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: waiting osd.2 seq 326417514498 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037772 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:40.374 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037772 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: waiting osd.3 seq 257698037772 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037772 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:40.375 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:40.376 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:40.376 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:40.376 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:40.376 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T16:04:40.376 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephte$seqs 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514499 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514499 2024-03-09T16:04:40.599 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: waiting osd.2 seq 326417514499 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037773 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037773 2024-03-09T16:04:40.600 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037773 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: waiting osd.3 seq 257698037773 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037773 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:40.601 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T16:04:40.602 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T16:04:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:41.835 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:04:41.836 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntust/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:42.029 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:42.030 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:04:42.031 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:42.678 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:194158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:42.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:42.856 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:194156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194155 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:04:42.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:04:42.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:04:42.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:04:42.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:04:42.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:04:42.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197076"' 2024-03-09T16:04:42.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197076' 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197078"' 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197078' 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 197076 197078' 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197076 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197079: /' 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:42.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197081: /' 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.693 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:04:45.694 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:45.695 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:45.696 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:45.696 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:45.696 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: start osd.1 2024-03-09T16:04:45.698 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:04:45.909 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.909 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:04:45.910 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:04:45.911 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:04:45.912 INFO:tasks.workunit.client.0.smithi146.stderr:197079: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:04:45.913 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:04:45.914 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:04:45.914 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:04:45.915 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:04:45.916 INFO:tasks.workunit.client.0.smithi146.stderr:197079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:04:45.917 INFO:tasks.workunit.client.0.smithi146.stderr:197079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:04:45.917 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:45.917 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:04:45.917 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:04:45.917 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:04:45.918 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:04:45.919 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:04:45.919 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:04:45.919 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:04:45.919 INFO:tasks.workunit.client.0.smithi146.stderr:197079: start osd.3 2024-03-09T16:04:45.919 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:04:50.282 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:04:50.282 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:04:50.282 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:50.282 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:50.282 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2024-03-09T16:04:45.726+0000 7f465ecd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2024-03-09T16:04:45.728+0000 7f465ecd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2024-03-09T16:04:45.729+0000 7f465ecd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 0 2024-03-09T16:04:50.283 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2024-03-09T16:04:46.252+0000 7f465ecd5640 -1 Falling back to public interface 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 1 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:50.284 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2024-03-09T16:04:48.866+0000 7f465ecd5640 -1 osd.1 80 log_to_monitors true 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 3 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.285 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197081: osd.1 up in weight 1 up_from 85 up_thru 85 down_at 81 last_197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2024-03-09T16:04:45.939+0000 7f8b2bb49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2024-03-09T16:04:45.941+0000 7f8b2bb49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2024-03-09T16:04:45.942+0000 7f8b2bb49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:04:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 0 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2024-03-09T16:04:46.465+0000 7f8b2bb49640 -1 Falling back to public interface 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 1 2024-03-09T16:04:50.517 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2024-03-09T16:04:49.068+0000 7f8b2bb49640 -1 osd.3 80 log_to_monitors true 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:04:50.518 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:04:50.519 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 3 2024-03-09T16:04:50.519 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:04:50.519 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197079: osd.3 up in weight 1 up_from 86 up_thru 86 down_at 81 last_clean_interval [77,80) [v2:127.0.0.1:6818/2793635137,v1:127.0.0.1:6819/2793635137] [v2:127.0.0.1:6820/2793635137,v1:127.0.0.1:6821/2793635137] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:51.061 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:51.062 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 1 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 2 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: 3' 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-365072220162' 2024-03-09T16:04:51.063 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [60,80) [v2:127.0.0.1:6826/3282877103,v1:127.0.0.1:6827/3282877103] [v2:127.0.0.1:6828/3282877103,v1:127.0.0.1:6829/3282877103] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:04:51.213 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 1 2024-03-09T16:04:51.214 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 2 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: 3' 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163' 2024-03-09T16:04:51.215 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:53.188 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:04:53.188 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514503 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514503 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-365072220162 2-326417514503' 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187458 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187458 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-365072220162 2-326417514503 3-369367187458' 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-03-09T16:04:53.189 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: waiting osd.0 seq 21474836542 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836542 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836542 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220162 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220162 2024-03-09T16:04:53.190 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: waiting osd.1 seq 365072220162 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-03-09T16:04:53.191 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:1970tell osd.2 flush_pg_stats 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514504 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514504 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163 2-326417514504' 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187459 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187459 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-365072220163 2-326417514504 3-369367187459' 2024-03-09T16:04:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: waiting osd.0 seq 21474836543 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836543 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:04:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220163 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220163 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220163' 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: waiting osd.1 seq 365072220163 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-03-09T16:04:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514503 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514503 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514503 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514503' 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: waiting osd.2 seq 326417514503 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:54.491 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514504 -lt 326417514503 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187458 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187458 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187458 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: waiting osd.3 seq 369367187458 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187458 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:54.492 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:04:54.493 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.c79: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514504 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514504 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514504 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514504' 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: waiting osd.2 seq 326417514504 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514504 -lt 326417514504 2024-03-09T16:04:54.550 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187459 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187459 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187459 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: waiting osd.3 seq 369367187459 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187459 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:04:54.551 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:04:54.552 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:56.012 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.012 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:56.013 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.014 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:56.015 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:56.016 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:04:56.016 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:56.067 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:56.068 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:04:56.069 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:04:56.070 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:04:56.071 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:04:56.071 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:56.771 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:197081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:56.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:04:56.848 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:197079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:04:56.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197078 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:04:56.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:04:56.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:04:56.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:57.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:57.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:04:21.932442+0000 2024-03-09T16:04:57.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:04:57.647 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:04:57.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:04:21.932442+0000 2024-03-09T16:04:57.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:04:21.932442+0000 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:57.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:57.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:57.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:57.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:57.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:04:21.932442+0000 '>' 2024-03-09T16:04:21.932442+0000 2024-03-09T16:04:57.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:04:58.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:04:58.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:04:58.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:04:58.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:04:58.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:04:58.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:04:58.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:04:59.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:04:21.932442+0000 '>' 2024-03-09T16:04:21.932442+0000 2024-03-09T16:04:59.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:05:00.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:05:00.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:05:00.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:05:00.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:05:00.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:05:00.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:05:00.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:05:00.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:04:58.025947+0000 '>' 2024-03-09T16:04:21.932442+0000 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200122"' 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200122' 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200124"' 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200124' 2024-03-09T16:05:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 200122 200124' 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200122 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200125: /' 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:05:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200127: /' 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.043 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:05:03.044 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: _ 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: hinfo_key 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: snapset 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:05:03.045 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:05:03.046 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:05:03.047 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:05:03.048 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:05:03.048 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:05:03.048 INFO:tasks.workunit.client.0.smithi146.stderr:200125: start osd.3 2024-03-09T16:05:03.395 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.395 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:05:03.395 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:05:03.395 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:05:03.395 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.396 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: _ 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: hinfo_key 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: snapset 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:05:03.397 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:03.398 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:05:03.399 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:05:03.400 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:05:03.400 INFO:tasks.workunit.client.0.smithi146.stderr:200127: start osd.1 2024-03-09T16:05:07.221 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2024-03-09T16:05:03.076+0000 7feddc5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2024-03-09T16:05:03.078+0000 7feddc5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2024-03-09T16:05:03.079+0000 7feddc5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:05:07.222 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 0 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2024-03-09T16:05:03.603+0000 7feddc5e1640 -1 Falling back to public interface 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 1 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:05:07.223 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2024-03-09T16:05:05.950+0000 7feddc5e1640 -1 osd.3 88 log_to_monitors true 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 3 2024-03-09T16:05:07.224 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:05:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2024-03-09T16:05:03.430+0000 7f17c8173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2024-03-09T16:05:03.465+0000 7f17c8173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2024-03-09T16:05:03.466+0000 7f17c8173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 0 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:08.098 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2024-03-09T16:05:03.994+0000 7f17c8173640 -1 Falling back to public interface 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 1 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:08.099 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2024-03-09T16:05:06.703+0000 7f17c8173640 -1 osd.1 88 log_to_monitors true 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 3 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:08.100 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: osd.3 up in weight 1 up_from 93 up_thru 86 down_at 89 last_clean_interval [86,88) [v2:127.0.0.1:6818/1460737483,v1:127.0.0.1:6819/1460737483] [v2:127.0.0.1:6820/1460737483,v1:127.0.0.1:6821/1460737483] exists,up bcaa5f20-4d03-44ec-a37f-b16fc3f98361 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:05:08.275 INFO:tasks.workunit.client.0.smithi146.stderr:200125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 1 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 2 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: 3' 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:05:08.276 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: Error ENXIO: problem getting command descriptions from osd.1 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-03-09T16:05:08.277 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-03-09T16:05:09.894 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_peph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:09.894 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:09.894 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 4 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: osd.1 up in weight 1 up_from 97 up_thru 97 down_at 89 last_clean_interval [85,88) [v2:127.0.0.1:6826/2129571681,v1:127.0.0.1:6827/2129571681] [v2:127.0.0.1:6828/2129571681,v1:127.0.0.1:6829/2129571681] exists,up 1b547d29-7a13-40fc-a5ce-32f7916a5837 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:05:09.895 INFO:tasks.workunit.client.0.smithi146.stderr:200127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 1 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 2 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: 3' 2024-03-09T16:05:09.896 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-03-09T16:05:09.897 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalong_stats: for osd in $ids 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514508 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514508 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 2-326417514508' 2024-03-09T16:05:10.795 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 2-326417514508 3-399431958530' 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: waiting osd.0 seq 21474836547 2024-03-09T16:05:10.796 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836547 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836547 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514508 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514508 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514508 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514508' 2024-03-09T16:05:10.797 INFO:tasks.workunit.client.0.smithi146.stderr:200125: waiting osd.2 seq 326417514508 2024-03-09T16:05:10.798 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:05:10.798 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514508 -lt 326417514508 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubue/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-416611827714' 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:10.987 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514510 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514510 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-416611827714 2-326417514510' 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-416611827714 2-326417514510 3-399431958531' 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:10.988 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-03-09T16:05:10.989 INFO:tasks.workunit.client.0.smithi146.stderr:200127: waiting osd.0 seq 21474836549 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-416611827714 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:10.990 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-416611827714 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 416611827714' 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: waiting osd.1 seq 416611827714 2024-03-09T16:05:10.991 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:05:12.164 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:12.164 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958530 2024-03-09T16:05:12.164 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:12.164 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958530 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: waiting osd.3 seq 399431958530 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:12.165 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:12.166 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:12.167 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:12.167 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:13.238 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hers.sh:2214: flush_pg_stats: test 1 -lt 416611827714 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827714 -lt 416611827714 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514510 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514510 2024-03-09T16:05:13.239 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514510 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514510' 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: waiting osd.2 seq 326417514510 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514510 -lt 326417514510 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958531 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958531 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:05:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: waiting osd.3 seq 399431958531 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958531 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains(elpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 1 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:05:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:13.602 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:05:13.602 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:05:13.602 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:05:14.643 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubunt"stale") | not)' 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:14.644 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:14.645 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:05:14.646 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:05:14.646 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:05:14.646 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:05:14.646 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:05:14.646 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200127: u/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:05:14.722 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:14.723 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:05:14.724 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:05:16.198 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recov//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:05:16.198 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:05:16.198 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-03-09T16:05:16.198 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:16.199 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:05:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:05:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_makery_progress: ceph --format json status 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:16.280 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:05:16.281 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:05:16.282 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:05:16.282 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:05:16.282 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:05:16.282 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T16:05:16.282 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T16:05:16.934 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/staning_recovery_progress: test 1 '!=' null 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:16.935 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:200127: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:05:16.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:05:17.075 INFO:tasks.workunit.client.0.smithi146.stderr:dalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:05:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:200125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:200125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200124 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:05:17.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:05:17.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:05:17.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:05:17.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:05:17.227 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:05:17.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:05:17.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:05:17.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:05:17.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:05:17.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:05:17.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:05:17.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:05:17.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:05:17.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:05:17.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:05:17.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:05:17.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:05:17.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:05:17.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:05:17.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:05:17.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:05:17.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:05:17.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:05:17.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:05:17.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:05:17.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:05:17.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:05:17.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:05:17.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:05:17.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:05:17.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:05:17.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:05:17.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:05:17.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:05:17.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:05:17.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:05:17.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:05:17.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:05:17.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:05:17.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:05:17.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:05:17.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:05:17.277 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:05:17.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:05:17.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:05:17.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:05:17.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:05:17.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:05:17.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:05:17.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:05:17.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:05:17.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:05:17.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:05:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:05:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:05:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:05:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:05:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.848 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.848 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:05:17.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:05:17.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:17.928 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:05:17.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:05:17.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:05:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:18.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:18.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:05:18.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:05:18.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:05:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:05:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:05:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:05:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7e30c6e-f888-4b7d-8757-4df118b51555 2024-03-09T16:05:18.863 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 e7e30c6e-f888-4b7d-8757-4df118b51555 2024-03-09T16:05:18.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7e30c6e-f888-4b7d-8757-4df118b51555' 2024-03-09T16:05:18.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:05:18.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+iOxlPDsRNBAAaogLi1w7lf1lNiE3g4cyhA== 2024-03-09T16:05:18.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+iOxlPDsRNBAAaogLi1w7lf1lNiE3g4cyhA=="}' 2024-03-09T16:05:18.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7e30c6e-f888-4b7d-8757-4df118b51555 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:05:19.077 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:05:19.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:05:19.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+iOxlPDsRNBAAaogLi1w7lf1lNiE3g4cyhA== --osd-uuid e7e30c6e-f888-4b7d-8757-4df118b51555 2024-03-09T16:05:19.117 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.116+0000 7f1b49e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:19.128 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.128+0000 7f1b49e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.129+0000 7f1b49e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:19.130 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.129+0000 7f1b49e0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:05:19.130 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.129+0000 7f1b49e0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:05:19.131 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.129+0000 7f1b49e0c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:05:19.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:19.219+0000 7f1b49e0c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:05:24.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:05:24.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:05:24.690 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:05:24.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:05:24.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:05:25.070 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:05:25.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:05:25.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:05:25.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:05:25.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:05:25.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:05:25.096 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:25.097+0000 7f4a788d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:25.128 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:25.128+0000 7f4a788d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:25.129 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:25.129+0000 7f4a788d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:25.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:05:25.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:05:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:25.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:25.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:25.656 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:25.657+0000 7f4a788d9640 -1 Falling back to public interface 2024-03-09T16:05:26.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:26.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:26.632 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:05:26.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:05:26.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:26.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:26.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:27.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:27.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:27.910 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:05:27.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:05:27.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:27.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:27.975 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:27.976+0000 7f4a788d9640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:05:28.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:29.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:29.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:29.186 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:05:29.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:05:29.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:29.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:29.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:30.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:30.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:30.471 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:05:30.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:05:30.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:30.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:30.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:31.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:31.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:31.771 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:05:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:05:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:32.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:33.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:33.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:33.052 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:05:33.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:05:33.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:33.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:33.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:34.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:34.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:34.331 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:05:34.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:05:34.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:34.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:34.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:35.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:35.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:35.610 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:05:35.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:05:35.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:35.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:35.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:36.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:36.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:36.905 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:05:36.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:05:36.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:36.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:37.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:38.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:38.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:38.187 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:05:38.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:05:38.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:38.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:38.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:39.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:39.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:39.471 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:05:39.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:05:39.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:39.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:40.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:41.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:41.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:41.210 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:05:41.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:05:41.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:41.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:41.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:42.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:42.719 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:05:42.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:42.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:05:42.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:42.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:43.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:44.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:44.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:44.712 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:05:44.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:05:44.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:44.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:45.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:46.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:46.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:46.114 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:05:46.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:05:46.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:46.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:46.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:47.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:47.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:47.531 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:05:47.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:05:47.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:47.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:48.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:49.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:49.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:49.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:05:49.153 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:05:49.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:49.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:49.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:50.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:50.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:50.838 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:05:50.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:05:50.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:50.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:05:51.182 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/474502083,v1:127.0.0.1:6803/474502083] [v2:127.0.0.1:6804/474502083,v1:127.0.0.1:6805/474502083] exists,up e7e30c6e-f888-4b7d-8757-4df118b51555 2024-03-09T16:05:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:05:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:05:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:05:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:05:51.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:05:51.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:05:51.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7bdabe3f-7d06-4036-ad8c-9729c995504c 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7bdabe3f-7d06-4036-ad8c-9729c995504c' 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 7bdabe3f-7d06-4036-ad8c-9729c995504c 2024-03-09T16:05:51.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:05:51.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfiOxlduMmDBAA0CGuRPsrg5+4RE3ZTTZivg== 2024-03-09T16:05:51.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfiOxlduMmDBAA0CGuRPsrg5+4RE3ZTTZivg=="}' 2024-03-09T16:05:51.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7bdabe3f-7d06-4036-ad8c-9729c995504c -i td/osd-scrub-repair/1/new.json 2024-03-09T16:05:51.587 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:05:51.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:05:51.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfiOxlduMmDBAA0CGuRPsrg5+4RE3ZTTZivg== --osd-uuid 7bdabe3f-7d06-4036-ad8c-9729c995504c 2024-03-09T16:05:51.628 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.628+0000 7f2a9f9e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:51.672 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.672+0000 7f2a9f9e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:51.675 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.673+0000 7f2a9f9e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:51.675 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.674+0000 7f2a9f9e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:05:51.675 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.674+0000 7f2a9f9e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:05:51.676 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.674+0000 7f2a9f9e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:05:51.747 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:51.747+0000 7f2a9f9e1640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:05:57.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:05:57.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:05:57.725 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:05:57.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:05:57.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:05:58.080 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:05:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:05:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:05:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:05:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:05:58.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:05:58.106 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:58.106+0000 7ff6e4d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:58.108 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:58.108+0000 7ff6e4d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:58.109 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:58.109+0000 7ff6e4d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:58.353 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:05:58.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:05:58.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:58.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:58.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:05:58.626 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:05:58.627+0000 7ff6e4d23640 -1 Falling back to public interface 2024-03-09T16:05:59.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:05:59.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:05:59.628 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:05:59.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:05:59.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:05:59.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:05:59.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:00.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:00.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:00.905 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:06:00.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:06:00.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:00.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:00.999 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:00.999+0000 7ff6e4d23640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:06:01.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:02.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:02.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:02.218 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:06:02.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:06:02.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:02.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:02.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:03.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:03.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:03.505 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:06:03.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:06:03.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:03.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:03.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:04.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:04.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:04.851 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:06:04.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:06:04.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:04.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:05.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:06.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:06.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:06.185 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:06:06.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:06:06.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:06.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:06.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:07.467 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:06:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:06:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:07.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:07.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:08.825 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:06:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:06:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:09.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:10.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:10.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:10.110 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:06:10.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:06:10.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:10.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:10.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:11.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:11.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:11.393 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:06:11.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:06:11.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:11.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:11.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:12.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:12.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:12.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:06:12.676 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:06:12.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:12.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:13.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:14.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:14.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:14.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:06:14.303 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:06:14.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:14.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:14.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:15.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:15.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:15.985 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:06:15.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:06:15.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:15.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:16.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:17.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:17.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:17.629 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:06:17.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:06:17.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:17.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:18.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:19.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:19.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:19.213 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:06:19.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:06:19.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:19.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:19.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:20.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:20.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:21.000 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:06:21.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:06:21.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:21.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:21.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:22.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:22.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:22.928 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:06:22.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:06:22.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:22.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:06:23.388 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4275030399,v1:127.0.0.1:6811/4275030399] [v2:127.0.0.1:6812/4275030399,v1:127.0.0.1:6813/4275030399] exists,up 7bdabe3f-7d06-4036-ad8c-9729c995504c 2024-03-09T16:06:23.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:06:23.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:06:23.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:06:23.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:06:23.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:06:23.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:06:23.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93ae9791-866c-41a4-b7b0-2bd4d24248bf 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 93ae9791-866c-41a4-b7b0-2bd4d24248bf' 2024-03-09T16:06:23.395 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 93ae9791-866c-41a4-b7b0-2bd4d24248bf 2024-03-09T16:06:23.396 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:06:23.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/iOxlODFwGBAAiz5kKsCadP0AW1/lDR5MqA== 2024-03-09T16:06:23.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/iOxlODFwGBAAiz5kKsCadP0AW1/lDR5MqA=="}' 2024-03-09T16:06:23.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93ae9791-866c-41a4-b7b0-2bd4d24248bf -i td/osd-scrub-repair/2/new.json 2024-03-09T16:06:23.736 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:06:23.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:06:23.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/iOxlODFwGBAAiz5kKsCadP0AW1/lDR5MqA== --osd-uuid 93ae9791-866c-41a4-b7b0-2bd4d24248bf 2024-03-09T16:06:23.776 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.775+0000 7ff344412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:23.822 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.820+0000 7ff344412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:23.822 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.821+0000 7ff344412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:23.823 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.822+0000 7ff344412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:06:23.823 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.822+0000 7ff344412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:06:23.823 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.822+0000 7ff344412640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:06:23.897 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:23.897+0000 7ff344412640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:06:30.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:06:30.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:06:30.137 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:06:30.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:06:30.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:06:30.490 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:06:30.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:06:30.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:06:30.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:06:30.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:06:30.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:06:30.517 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:30.516+0000 7f0cd09fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:30.518 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:30.518+0000 7f0cd09fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:30.519 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:30.519+0000 7f0cd09fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:30.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:06:30.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:06:30.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:06:30.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:30.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:31.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:31.038 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:31.038+0000 7f0cd09fa640 -1 Falling back to public interface 2024-03-09T16:06:32.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:32.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:32.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:06:32.039 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:06:32.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:32.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:32.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:33.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:33.315 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:06:33.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:33.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:06:33.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:33.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:33.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:33.434+0000 7f0cd09fa640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:06:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:34.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:34.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:34.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:06:34.636 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:06:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:34.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:34.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:35.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:35.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:35.919 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:06:35.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:06:35.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:35.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:36.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:37.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:37.207 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:06:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:06:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:37.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:38.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:38.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:38.507 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:06:38.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:06:38.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:38.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:38.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:39.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:39.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:39.790 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:06:39.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:06:39.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:39.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:40.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:41.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:41.075 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:06:41.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:41.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:06:41.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:41.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:41.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:42.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:42.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:42.450 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:06:42.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:06:42.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:42.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:42.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:43.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:43.799 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:06:43.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:43.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:06:43.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:43.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:44.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:45.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:45.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:45.195 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:06:45.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:06:45.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:45.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:45.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:46.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:46.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:46.621 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:06:46.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:06:46.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:46.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:47.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:48.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:48.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:48.048 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:06:48.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:06:48.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:48.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:48.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:49.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:49.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:49.599 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:06:49.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:06:49.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:49.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:50.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:51.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:51.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:51.296 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:06:51.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:06:51.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:51.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:51.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:52.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:52.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:52.954 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:06:52.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:06:52.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:52.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:53.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:54.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:54.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:54.390 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:06:54.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:06:54.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:54.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:54.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:06:55.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:06:55.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:06:55.893 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:06:55.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:06:55.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:06:55.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:06:56.245 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3201245851,v1:127.0.0.1:6819/3201245851] [v2:127.0.0.1:6820/3201245851,v1:127.0.0.1:6821/3201245851] exists,up 93ae9791-866c-41a4-b7b0-2bd4d24248bf 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:06:56.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:06:56.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:06:56.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:06:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 fc1f68e9-3c93-4488-b56c-b09f36aba9ea' 2024-03-09T16:06:56.253 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:06:56.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:06:56.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgiexlqD3nDxAAqO+0BSRJ84xqi8NXmoDcrQ== 2024-03-09T16:06:56.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgiexlqD3nDxAAqO+0BSRJ84xqi8NXmoDcrQ=="}' 2024-03-09T16:06:56.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc1f68e9-3c93-4488-b56c-b09f36aba9ea -i td/osd-scrub-repair/3/new.json 2024-03-09T16:06:56.736 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:06:56.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T16:06:56.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgiexlqD3nDxAAqO+0BSRJ84xqi8NXmoDcrQ== --osd-uuid fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:06:56.777 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.777+0000 7f0b91412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.805+0000 7f0b91412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.806+0000 7f0b91412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:06:56.806 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.806+0000 7f0b91412640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:06:56.806 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.806+0000 7f0b91412640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:06:56.806 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.806+0000 7f0b91412640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:06:56.895 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:06:56.895+0000 7f0b91412640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T16:07:02.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T16:07:02.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:07:02.972 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T16:07:02.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T16:07:02.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:07:03.326 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T16:07:03.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T16:07:03.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:07:03.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:07:03.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:07:03.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:07:03.352 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:03.351+0000 7f836acce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:03.353 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:03.354+0000 7f836acce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:03.354 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:03.355+0000 7f836acce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:03.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:07:03.601 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:07:03.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:03.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:03.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:03.887 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:03.887+0000 7f836acce640 -1 Falling back to public interface 2024-03-09T16:07:04.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:04.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:04.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:07:04.874 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:07:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:05.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:06.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:06.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:06.154 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:07:06.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:07:06.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:06.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:06.342 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:06.342+0000 7f836acce640 -1 osd.3 0 log_to_monitors true 2024-03-09T16:07:06.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:07.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:07.441 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:07:07.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:07.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:07:07.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:07.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:07.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:08.724 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:07:08.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:07:08.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:08.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:09.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:10.012 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:07:10.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:07:10.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:10.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:10.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:11.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:11.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:11.354 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:07:11.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:07:11.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:11.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:11.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:12.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:12.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:12.637 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:07:12.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:07:12.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:12.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:12.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:13.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:13.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:13.919 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:07:13.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:07:13.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:13.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:14.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:15.204 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:07:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:07:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:15.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:16.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:16.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:16.489 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:07:16.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:07:16.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:16.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:16.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:17.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:17.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:17.771 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:07:17.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:07:17.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:17.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:18.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:19.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:19.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:19.178 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:07:19.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:07:19.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:19.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:19.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:20.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:20.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:20.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:07:20.807 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:07:20.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:20.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:21.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:22.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:22.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:22.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:07:22.448 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:07:22.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:22.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:23.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:24.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:24.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:24.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:07:24.047 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:07:24.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:24.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:24.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:25.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:25.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:25.327 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:07:25.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:07:25.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:25.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:25.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:26.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:26.744 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:07:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:07:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:26.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:27.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:28.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:28.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:28.221 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:07:28.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:07:28.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:28.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:28.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:29.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:29.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:29.584 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:07:29.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:07:29.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:29.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:07:29.862 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1294051989,v1:127.0.0.1:6827/1294051989] [v2:127.0.0.1:6828/1294051989,v1:127.0.0.1:6829/1294051989] exists,up fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:07:29.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:07:29.868 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:07:29.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=68a68843-e8c1-477a-b343-bc2d09668ebc 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 68a68843-e8c1-477a-b343-bc2d09668ebc' 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stdout:add osd4 68a68843-e8c1-477a-b343-bc2d09668ebc 2024-03-09T16:07:29.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:07:29.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBiexlzCi9NBAAJIfAmhz9KeOmcL1ZMbBwFw== 2024-03-09T16:07:29.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBiexlzCi9NBAAJIfAmhz9KeOmcL1ZMbBwFw=="}' 2024-03-09T16:07:29.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 68a68843-e8c1-477a-b343-bc2d09668ebc -i td/osd-scrub-repair/4/new.json 2024-03-09T16:07:30.242 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:07:30.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-09T16:07:30.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBiexlzCi9NBAAJIfAmhz9KeOmcL1ZMbBwFw== --osd-uuid 68a68843-e8c1-477a-b343-bc2d09668ebc 2024-03-09T16:07:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.281+0000 7fec9e993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:30.318 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.317+0000 7fec9e993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.318+0000 7fec9e993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.318+0000 7fec9e993640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:07:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.319+0000 7fec9e993640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:07:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.319+0000 7fec9e993640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:07:30.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:30.401+0000 7fec9e993640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-09T16:07:36.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-09T16:07:36.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:07:36.733 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd4 key to auth repository 2024-03-09T16:07:36.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T16:07:36.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:07:37.090 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.4 2024-03-09T16:07:37.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T16:07:37.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:07:37.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:07:37.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:07:37.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:07:37.113 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:37.112+0000 7f1c5160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:37.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:37.115+0000 7f1c5160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:37.117 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:37.116+0000 7f1c5160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:07:37.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T16:07:37.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:07:37.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T16:07:37.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:07:37.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:07:37.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:37.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:07:37.378 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:07:37.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:37.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:37.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:37.657 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:37.657+0000 7f1c5160f640 -1 Falling back to public interface 2024-03-09T16:07:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:38.650 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:07:38.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:07:38.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:38.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:38.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:39.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:39.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:39.929 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:07:39.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:07:39.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:39.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:40.075 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:07:40.074+0000 7f1c5160f640 -1 osd.4 0 log_to_monitors true 2024-03-09T16:07:40.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:41.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:41.207 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:07:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:07:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:41.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:41.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:42.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:42.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:42.489 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:07:42.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:07:42.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:42.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:42.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:43.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:43.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:43.820 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:07:43.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:07:43.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:43.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:44.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:45.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:45.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:45.164 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:07:45.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:07:45.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:45.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:45.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:46.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:46.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:46.445 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:07:46.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:07:46.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:46.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:46.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:47.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:47.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:47.729 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:07:47.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:07:47.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:47.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:48.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:49.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:49.015 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:07:49.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:49.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:07:49.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:49.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:49.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:50.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:50.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:50.331 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:07:50.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:07:50.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:50.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:50.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:51.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:51.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:51.616 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:07:51.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:07:51.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:51.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:51.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:52.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:52.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:52.999 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:07:52.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:07:53.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:53.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:53.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:54.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:54.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:54.839 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:07:54.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:07:54.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:54.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:55.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:56.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:56.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:56.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:07:56.879 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:07:56.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:56.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:57.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:07:58.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:07:58.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:07:58.632 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:07:58.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:07:58.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:07:58.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:07:59.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:00.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:00.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:00.211 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:08:00.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:08:00.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:00.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:08:00.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:01.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:01.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:01.971 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:08:01.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:08:01.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:01.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:08:02.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:03.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:03.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:03.539 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:08:03.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:08:03.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:03.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:08:03.819 INFO:tasks.workunit.client.0.smithi146.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/808316556,v1:127.0.0.1:6835/808316556] [v2:127.0.0.1:6836/808316556,v1:127.0.0.1:6837/808316556] exists,up 68a68843-e8c1-477a-b343-bc2d09668ebc 2024-03-09T16:08:03.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:08:03.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:08:03.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:08:03.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:08:03.824 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:08:03.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:08:03.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:08:03.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:08:03.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ecf7f5db-2c6b-400d-8c88-a4d3c606d8da' 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stdout:add osd5 ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:08:03.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:08:03.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjiexlWMAcMhAAqdFu+oD7kztubCbkdFcR7g== 2024-03-09T16:08:03.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjiexlWMAcMhAAqdFu+oD7kztubCbkdFcR7g=="}' 2024-03-09T16:08:03.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ecf7f5db-2c6b-400d-8c88-a4d3c606d8da -i td/osd-scrub-repair/5/new.json 2024-03-09T16:08:04.273 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:08:04.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-09T16:08:04.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjiexlWMAcMhAAqdFu+oD7kztubCbkdFcR7g== --osd-uuid ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:08:04.313 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.313+0000 7fcf17fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:04.367 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.367+0000 7fcf17fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:04.367 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.368+0000 7fcf17fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:04.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.368+0000 7fcf17fdd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:08:04.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.368+0000 7fcf17fdd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:08:04.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.368+0000 7fcf17fdd640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:08:04.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:04.500+0000 7fcf17fdd640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-09T16:08:10.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-09T16:08:10.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:08:10.666 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd5 key to auth repository 2024-03-09T16:08:10.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T16:08:10.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:08:11.017 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.5 2024-03-09T16:08:11.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T16:08:11.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:08:11.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:08:11.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:08:11.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:08:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:11.044+0000 7ffb7020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:11.045 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:11.046+0000 7ffb7020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:11.046 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:11.047+0000 7ffb7020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:08:11.290 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:08:11.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:11.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:11.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:11.562 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:11.562+0000 7ffb7020c640 -1 Falling back to public interface 2024-03-09T16:08:12.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:12.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:12.563 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:08:12.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:08:12.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:12.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:12.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:13.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:13.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:13.843 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:08:13.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:08:13.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:13.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:13.875+0000 7ffb7020c640 -1 osd.5 0 log_to_monitors true 2024-03-09T16:08:14.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:15.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:15.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:15.185 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:08:15.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:08:15.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:15.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:15.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:16.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:16.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:16.470 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:08:16.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:08:16.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:16.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:16.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:17.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:17.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:17.836 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:08:17.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:08:17.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:17.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:18.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:19.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:19.183 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:08:19.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:19.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:08:19.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:19.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:19.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:20.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:20.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:20.469 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:08:20.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:08:20.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:20.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:20.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:21.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:21.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:21.862 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:08:21.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:08:21.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:21.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:22.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:23.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:23.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:23.193 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:08:23.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:08:23.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:23.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:23.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:24.476 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:08:24.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:08:24.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:24.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:24.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:25.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:25.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:25.819 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:08:25.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:08:25.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:25.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:26.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:27.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:27.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:27.150 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:08:27.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:08:27.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:27.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:27.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:28.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:28.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:28.432 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:08:28.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:08:28.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:28.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:28.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:29.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:29.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:29.995 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:08:29.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:08:29.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:29.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:30.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:31.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:31.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:31.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:08:31.756 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:08:31.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:31.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:32.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:33.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:33.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:33.534 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:08:33.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:08:33.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:33.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:34.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:35.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:35.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:35.172 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:08:35.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:08:35.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:35.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:36.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:36.947 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:08:36.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:36.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:08:36.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:36.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:37.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:38.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:38.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:38.226 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:08:38.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:08:38.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:38.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:08:38.573 INFO:tasks.workunit.client.0.smithi146.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4230120661,v1:127.0.0.1:6843/4230120661] [v2:127.0.0.1:6844/4230120661,v1:127.0.0.1:6845/4230120661] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:08:38.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:08:38.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:08:38.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:08:38.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:08:38.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:08:38.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:08:38.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:08:38.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:08:38.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37516937-de5e-434c-8745-5b4c5307fb88 2024-03-09T16:08:38.581 INFO:tasks.workunit.client.0.smithi146.stdout:add osd6 37516937-de5e-434c-8745-5b4c5307fb88 2024-03-09T16:08:38.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 37516937-de5e-434c-8745-5b4c5307fb88' 2024-03-09T16:08:38.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:08:38.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGiexlP9V0IxAABgC4YRumcaDMAY1DX5ld2g== 2024-03-09T16:08:38.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGiexlP9V0IxAABgC4YRumcaDMAY1DX5ld2g=="}' 2024-03-09T16:08:38.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37516937-de5e-434c-8745-5b4c5307fb88 -i td/osd-scrub-repair/6/new.json 2024-03-09T16:08:38.952 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:08:38.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-09T16:08:38.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGiexlP9V0IxAABgC4YRumcaDMAY1DX5ld2g== --osd-uuid 37516937-de5e-434c-8745-5b4c5307fb88 2024-03-09T16:08:38.991 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:38.991+0000 7f8beec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:39.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.037+0000 7f8beec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:39.037 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.038+0000 7f8beec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:39.038 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.038+0000 7f8beec11640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:08:39.038 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.038+0000 7f8beec11640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:08:39.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.038+0000 7f8beec11640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:08:39.112 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:39.112+0000 7f8beec11640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-09T16:08:45.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-09T16:08:45.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:08:45.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T16:08:45.366 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd6 key to auth repository 2024-03-09T16:08:45.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:08:45.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T16:08:45.718 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.6 2024-03-09T16:08:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:08:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:08:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:08:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:08:45.743 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:45.742+0000 7fabfb5dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:45.745 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:45.744+0000 7fabfb5dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:45.746 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:45.745+0000 7fabfb5dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:08:46.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T16:08:46.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:08:46.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T16:08:46.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:08:46.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:08:46.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:46.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:08:46.018 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:08:46.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:46.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:46.271 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:46.270+0000 7fabfb5dd640 -1 Falling back to public interface 2024-03-09T16:08:46.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:47.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:47.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:47.293 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:08:47.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:08:47.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:47.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:47.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:48.570 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:08:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:08:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:48.613 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:08:48.612+0000 7fabfb5dd640 -1 osd.6 0 log_to_monitors true 2024-03-09T16:08:48.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:49.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:49.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:49.850 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:08:49.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:08:49.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:49.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:50.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:51.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:51.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:51.223 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:08:51.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:08:51.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:51.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:51.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:52.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:52.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:52.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:08:52.590 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:08:52.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:52.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:52.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:53.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:53.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:53.874 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:08:53.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:08:53.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:53.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:54.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:55.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:55.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:55.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:08:55.214 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:08:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:55.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:56.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:56.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:56.528 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:08:56.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:08:56.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:56.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:56.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:57.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:57.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:57.808 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:08:57.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:08:57.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:57.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:58.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:08:59.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:08:59.203 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:08:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:08:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:08:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:08:59.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:08:59.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:00.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:00.972 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:09:00.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:00.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:09:00.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:00.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:00.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:01.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:01.770 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:09:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:09:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:02.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:03.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:03.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:03.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:09:03.262 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:09:03.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:03.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:03.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:04.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:04.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:04.697 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:09:04.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:09:04.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:04.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:05.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:06.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:06.041 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:09:06.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:06.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:09:06.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:06.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:06.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:07.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:07.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:07.727 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:09:07.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:09:07.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:07.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:08.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:09.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:09.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:09.539 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:09:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:09:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:10.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:11.181 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:09:11.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:11.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:09:11.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:11.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:11.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:12.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:12.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:12.689 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:09:12.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:09:12.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:12.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:09:12.971 INFO:tasks.workunit.client.0.smithi146.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3665959730,v1:127.0.0.1:6851/3665959730] [v2:127.0.0.1:6852/3665959730,v1:127.0.0.1:6853/3665959730] exists,up 37516937-de5e-434c-8745-5b4c5307fb88 2024-03-09T16:09:12.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:09:12.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-09T16:09:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:09:12.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:09:12.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:09:12.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:09:12.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:09:12.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9d973db-2b40-4dc6-8883-9d3ea4261c7e 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 d9d973db-2b40-4dc6-8883-9d3ea4261c7e' 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stdout:add osd7 d9d973db-2b40-4dc6-8883-9d3ea4261c7e 2024-03-09T16:09:12.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:09:12.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoiexlyYEsOxAA9LVUDUFkvXQZKSRsNPHh/w== 2024-03-09T16:09:12.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoiexlyYEsOxAA9LVUDUFkvXQZKSRsNPHh/w=="}' 2024-03-09T16:09:12.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9d973db-2b40-4dc6-8883-9d3ea4261c7e -i td/osd-scrub-repair/7/new.json 2024-03-09T16:09:13.370 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:09:13.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-09T16:09:13.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoiexlyYEsOxAA9LVUDUFkvXQZKSRsNPHh/w== --osd-uuid d9d973db-2b40-4dc6-8883-9d3ea4261c7e 2024-03-09T16:09:13.411 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.410+0000 7f155a38a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:13.455 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.455+0000 7f155a38a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:13.455 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.456+0000 7f155a38a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:13.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.456+0000 7f155a38a640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:09:13.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.456+0000 7f155a38a640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:09:13.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.456+0000 7f155a38a640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:09:13.521 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:13.521+0000 7f155a38a640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-09T16:09:19.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-09T16:09:19.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:09:19.738 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd7 key to auth repository 2024-03-09T16:09:19.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T16:09:19.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:09:20.094 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.7 2024-03-09T16:09:20.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T16:09:20.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:09:20.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:09:20.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:09:20.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:09:20.120 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:20.120+0000 7f67d2de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:20.122 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:20.122+0000 7f67d2de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:20.123 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:20.123+0000 7f67d2de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:09:20.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:20.378 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:09:20.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:09:20.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:20.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:20.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:20.663 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:20.663+0000 7f67d2de1640 -1 Falling back to public interface 2024-03-09T16:09:21.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:21.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:21.651 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:09:21.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:09:21.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:21.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:21.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:22.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:22.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:22.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:09:22.932 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:09:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:23.055 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:23.055+0000 7f67d2de1640 -1 osd.7 0 log_to_monitors true 2024-03-09T16:09:23.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:24.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:24.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:24.291 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:09:24.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:09:24.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:24.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:24.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:25.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:25.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:25.600 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:09:25.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:09:25.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:25.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:26.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:27.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:27.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:27.025 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:09:27.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:09:27.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:27.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:27.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:28.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:28.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:28.309 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:09:28.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:09:28.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:28.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:28.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:29.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:29.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:29.594 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:09:29.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:09:29.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:29.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:29.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:30.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:30.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:30.943 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:09:30.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:09:30.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:30.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:31.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:32.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:32.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:32.280 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:09:32.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:09:32.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:32.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:32.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:33.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:33.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:33.564 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:09:33.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:09:33.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:33.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:33.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:34.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:34.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:34.899 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:09:34.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:09:34.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:34.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:35.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:36.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:36.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:36.861 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:09:36.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:09:36.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:36.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:37.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:38.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:38.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:38.271 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:09:38.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:09:38.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:38.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:38.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:39.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:39.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:39.859 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:09:39.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:09:39.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:39.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:40.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:41.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:41.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:41.412 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:09:41.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:09:41.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:41.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:42.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:43.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:43.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:43.431 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:09:43.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:09:43.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:43.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:43.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:44.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:44.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:44.863 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:09:44.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:09:44.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:44.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:45.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:46.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:46.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:46.281 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:09:46.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:09:46.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:46.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:09:46.562 INFO:tasks.workunit.client.0.smithi146.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/640693333,v1:127.0.0.1:6859/640693333] [v2:127.0.0.1:6860/640693333,v1:127.0.0.1:6861/640693333] exists,up d9d973db-2b40-4dc6-8883-9d3ea4261c7e 2024-03-09T16:09:46.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:09:46.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:09:46.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e33bab14-8a79-405a-b1f9-780a97353f4d 2024-03-09T16:09:46.569 INFO:tasks.workunit.client.0.smithi146.stdout:add osd8 e33bab14-8a79-405a-b1f9-780a97353f4d 2024-03-09T16:09:46.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 e33bab14-8a79-405a-b1f9-780a97353f4d' 2024-03-09T16:09:46.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:09:46.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKiexlzNDOIhAAPFu9H9gWjhK4p6Ie0kJf8w== 2024-03-09T16:09:46.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKiexlzNDOIhAAPFu9H9gWjhK4p6Ie0kJf8w=="}' 2024-03-09T16:09:46.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e33bab14-8a79-405a-b1f9-780a97353f4d -i td/osd-scrub-repair/8/new.json 2024-03-09T16:09:46.931 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:09:46.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-09T16:09:46.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKiexlzNDOIhAAPFu9H9gWjhK4p6Ie0kJf8w== --osd-uuid e33bab14-8a79-405a-b1f9-780a97353f4d 2024-03-09T16:09:46.971 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:46.971+0000 7ff70d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:47.007 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.007+0000 7ff70d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:47.008 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.008+0000 7ff70d019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:47.009 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.009+0000 7ff70d019640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:09:47.009 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.009+0000 7ff70d019640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:09:47.009 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.009+0000 7ff70d019640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:09:47.091 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:47.091+0000 7ff70d019640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-09T16:09:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-09T16:09:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:09:53.287 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd8 key to auth repository 2024-03-09T16:09:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-09T16:09:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:09:53.635 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.8 2024-03-09T16:09:53.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-09T16:09:53.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:09:53.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:09:53.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:09:53.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:09:53.662 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:53.661+0000 7f3ed176f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:53.664 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:53.663+0000 7f3ed176f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:53.665 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:53.664+0000 7f3ed176f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:09:53.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:09:53.911 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:09:53.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:53.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:09:54.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:54.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:54.193+0000 7f3ed176f640 -1 Falling back to public interface 2024-03-09T16:09:55.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:55.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:55.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:09:55.184 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:09:55.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:55.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:09:55.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:56.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:56.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:56.469 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:09:56.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:09:56.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:56.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:09:56.477 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:09:56.476+0000 7f3ed176f640 -1 osd.8 0 log_to_monitors true 2024-03-09T16:09:56.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:57.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:57.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:57.753 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:09:57.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:09:57.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:57.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:09:58.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:09:59.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:09:59.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:09:59.041 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:09:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:09:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:09:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:09:59.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:00.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:00.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:00.394 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:10:00.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:10:00.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:00.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:00.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:01.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:01.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:01.772 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:10:01.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:10:01.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:01.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:02.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:03.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:03.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:03.054 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:10:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:10:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:04.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:04.380 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:10:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:10:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:04.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:04.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:05.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:05.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:05.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:10:05.696 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:10:05.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:05.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:05.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:07.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:07.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:07.000 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:10:07.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:10:07.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:07.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:07.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:08.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:08.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:08.280 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:10:08.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:10:08.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:08.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:08.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:09.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:09.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:09.787 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:10:09.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:10:09.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:09.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:10.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:11.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:11.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:11.264 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:10:11.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:10:11.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:11.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:12.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:13.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:13.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:13.026 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:10:13.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:10:13.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:13.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:13.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:14.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:14.383 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:10:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:10:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:14.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:14.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:15.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:15.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:15.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:10:15.886 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:10:15.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:15.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:16.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:17.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:17.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:17.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:10:17.299 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:10:17.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:17.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:17.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:18.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:18.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:18.592 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:10:18.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:10:18.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:18.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:18.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:19.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:19.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:19.876 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:10:19.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:10:19.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:19.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:10:20.180 INFO:tasks.workunit.client.0.smithi146.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/381269491,v1:127.0.0.1:6867/381269491] [v2:127.0.0.1:6868/381269491,v1:127.0.0.1:6869/381269491] exists,up e33bab14-8a79-405a-b1f9-780a97353f4d 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-09T16:10:20.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:10:20.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:10:20.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stdout:add osd9 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:10:20.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 74620015-395d-484e-afe7-a1d35ddc2ca4' 2024-03-09T16:10:20.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:10:20.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsiexluGr6CxAAPNXq3BpvpUyNPyt+tLDFJQ== 2024-03-09T16:10:20.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsiexluGr6CxAAPNXq3BpvpUyNPyt+tLDFJQ=="}' 2024-03-09T16:10:20.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74620015-395d-484e-afe7-a1d35ddc2ca4 -i td/osd-scrub-repair/9/new.json 2024-03-09T16:10:20.560 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:10:20.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-09T16:10:20.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsiexluGr6CxAAPNXq3BpvpUyNPyt+tLDFJQ== --osd-uuid 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:10:20.600 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.600+0000 7f4053e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:20.636 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.636+0000 7f4053e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:20.637 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.637+0000 7f4053e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:20.637 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.637+0000 7f4053e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:10:20.637 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.637+0000 7f4053e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:10:20.638 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.637+0000 7f4053e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:10:20.714 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:20.714+0000 7f4053e0e640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-09T16:10:26.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-09T16:10:26.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:10:26.356 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd9 key to auth repository 2024-03-09T16:10:26.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-09T16:10:26.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:10:26.705 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.9 2024-03-09T16:10:26.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-09T16:10:26.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:10:26.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:10:26.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:10:26.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:10:26.731 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:26.731+0000 7f6982caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:26.733 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:26.733+0000 7f6982caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:26.734 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:26.734+0000 7f6982caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:10:26.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:10:26.977 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:10:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:27.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:27.255 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:27.255+0000 7f6982caa640 -1 Falling back to public interface 2024-03-09T16:10:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:28.250 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:10:28.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:10:28.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:28.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:28.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:29.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:29.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:29.530 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:10:29.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:10:29.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:29.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:29.599 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:10:29.599+0000 7f6982caa640 -1 osd.9 0 log_to_monitors true 2024-03-09T16:10:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:30.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:30.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:30.816 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:10:30.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:10:30.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:30.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:31.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:32.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:32.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:32.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:10:32.102 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:10:32.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:32.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:32.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:33.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:33.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:33.416 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:10:33.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:10:33.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:33.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:33.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:34.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:34.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:34.699 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:10:34.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:10:34.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:34.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:34.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:35.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:35.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:35.979 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:10:35.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:10:35.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:35.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:36.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:37.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:37.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:37.265 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:10:37.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:10:37.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:37.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:37.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:38.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:38.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:38.549 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:10:38.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:10:38.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:38.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:38.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:39.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:39.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:39.837 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:10:39.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:10:39.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:39.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:40.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:41.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:41.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:41.255 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:10:41.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:10:41.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:41.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:42.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:43.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:43.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:43.227 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:10:43.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:10:43.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:43.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:43.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:44.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:44.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:44.913 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:10:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:10:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:45.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:46.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:46.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:46.540 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:10:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:10:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:46.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:47.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:48.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:48.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:48.206 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:10:48.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:10:48.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:48.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:49.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:49.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:49.588 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:10:49.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:10:49.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:49.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:50.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:51.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:51.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:51.551 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:10:51.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:10:51.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:51.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:52.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:10:53.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:10:53.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:10:53.446 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:10:53.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:10:53.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:10:53.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:10:53.719 INFO:tasks.workunit.client.0.smithi146.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3301765727,v1:127.0.0.1:6875/3301765727] [v2:127.0.0.1:6876/3301765727,v1:127.0.0.1:6877/3301765727] exists,up 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:10:53.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:10:53.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:10:53.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:10:53.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-09T16:10:53.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:10:53.999 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:10:54.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:10:54.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:10:54.544 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:10:54.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:10:55.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:10:58.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-09T16:10:58.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:10:58.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:10:58.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:10:58.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:10:58.405 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:10:58.405 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:10:58.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:10:58.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:10:58.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:10:58.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:10:58.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:10:58.939 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:10:58.939 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:58.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:10:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-03-09T16:10:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-03-09T16:10:59.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-03-09T16:10:59.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:10:59.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-03-09T16:10:59.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-03-09T16:10:59.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020' 2024-03-09T16:10:59.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.148 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:10:59.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-03-09T16:10:59.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-03-09T16:10:59.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491' 2024-03-09T16:10:59.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:10:59.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345969 2024-03-09T16:10:59.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345969 2024-03-09T16:10:59.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969' 2024-03-09T16:10:59.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:10:59.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182439 2024-03-09T16:10:59.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182439 2024-03-09T16:10:59.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439' 2024-03-09T16:10:59.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:10:59.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-03-09T16:10:59.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-03-09T16:10:59.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439 5-128849018914' 2024-03-09T16:10:59.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:10:59.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855385 2024-03-09T16:10:59.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855385 2024-03-09T16:10:59.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439 5-128849018914 6-150323855385' 2024-03-09T16:10:59.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:10:59.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-03-09T16:10:59.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-03-09T16:10:59.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439 5-128849018914 6-150323855385 7-171798691864' 2024-03-09T16:10:59.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:10:59.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528335 2024-03-09T16:10:59.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528335 2024-03-09T16:10:59.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439 5-128849018914 6-150323855385 7-171798691864 8-193273528335' 2024-03-09T16:10:59.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:10:59.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673020 2-64424509491 3-85899345969 4-107374182439 5-128849018914 6-150323855385 7-171798691864 8-193273528335 9-214748364806' 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-03-09T16:10:59.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836544 2024-03-09T16:10:59.980 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:00.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836544 2024-03-09T16:11:00.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:11:01.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:11:01.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:01.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-03-09T16:11:01.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:01.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-03-09T16:11:01.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:01.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:11:01.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-03-09T16:11:01.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-03-09T16:11:01.541 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949673020 2024-03-09T16:11:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-03-09T16:11:01.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:11:01.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-03-09T16:11:01.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:01.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-03-09T16:11:01.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:01.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:11:01.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-03-09T16:11:01.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:01.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-03-09T16:11:01.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-03-09T16:11:01.816 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509491 2024-03-09T16:11:01.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:11:02.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509491 -lt 64424509491 2024-03-09T16:11:02.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:02.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345969 2024-03-09T16:11:02.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:02.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:11:02.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345969 2024-03-09T16:11:02.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:02.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345969 2024-03-09T16:11:02.096 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345969 2024-03-09T16:11:02.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345969' 2024-03-09T16:11:02.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:02.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345970 -lt 85899345969 2024-03-09T16:11:02.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:02.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182439 2024-03-09T16:11:02.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:02.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:11:02.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182439 2024-03-09T16:11:02.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:02.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182439 2024-03-09T16:11:02.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-03-09T16:11:02.372 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.4 seq 107374182439 2024-03-09T16:11:02.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:02.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182439 2024-03-09T16:11:02.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:02.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018914 2024-03-09T16:11:02.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:02.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:11:02.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018914 2024-03-09T16:11:02.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:02.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-03-09T16:11:02.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-03-09T16:11:02.653 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.5 seq 128849018914 2024-03-09T16:11:02.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:11:02.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-03-09T16:11:02.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855385 2024-03-09T16:11:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:02.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:11:02.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855385 2024-03-09T16:11:02.975 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:02.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855385 2024-03-09T16:11:02.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855385' 2024-03-09T16:11:02.976 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.6 seq 150323855385 2024-03-09T16:11:02.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:11:03.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855386 -lt 150323855385 2024-03-09T16:11:03.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:03.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-03-09T16:11:03.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:03.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:11:03.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-03-09T16:11:03.299 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:03.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-03-09T16:11:03.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-03-09T16:11:03.300 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.7 seq 171798691864 2024-03-09T16:11:03.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:11:03.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-03-09T16:11:03.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:03.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528335 2024-03-09T16:11:03.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:11:03.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528335 2024-03-09T16:11:03.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:03.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528335 2024-03-09T16:11:03.577 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.8 seq 193273528335 2024-03-09T16:11:03.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-03-09T16:11:03.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:11:03.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528335 2024-03-09T16:11:03.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:03.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364806 2024-03-09T16:11:03.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:03.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:11:03.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364806 2024-03-09T16:11:03.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:03.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-03-09T16:11:03.855 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.9 seq 214748364806 2024-03-09T16:11:03.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2024-03-09T16:11:03.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:11:04.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-03-09T16:11:04.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:11:04.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:04.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:04.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:11:04.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:11:04.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:11:04.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:11:04.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:04.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:11:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-09T16:11:05.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:11:05.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:11:06.230 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:11:06.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:11:07.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:11:07.245 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:11:07.245 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:11:07.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:11:07.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:11:07.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:11:07.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:11:07.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:11:07.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:07.718 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:11:07.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-03-09T16:11:07.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-03-09T16:11:07.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-03-09T16:11:07.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:07.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:11:07.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-03-09T16:11:07.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-03-09T16:11:07.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022' 2024-03-09T16:11:07.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:07.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:11:08.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-03-09T16:11:08.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-03-09T16:11:08.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494' 2024-03-09T16:11:08.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:11:08.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345972 2024-03-09T16:11:08.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345972 2024-03-09T16:11:08.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972' 2024-03-09T16:11:08.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:11:08.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182441 2024-03-09T16:11:08.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182441 2024-03-09T16:11:08.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441' 2024-03-09T16:11:08.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:11:08.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018917 2024-03-09T16:11:08.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018917 2024-03-09T16:11:08.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441 5-128849018917' 2024-03-09T16:11:08.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:11:08.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-03-09T16:11:08.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-03-09T16:11:08.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441 5-128849018917 6-150323855388' 2024-03-09T16:11:08.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:11:08.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-03-09T16:11:08.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-03-09T16:11:08.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441 5-128849018917 6-150323855388 7-171798691867' 2024-03-09T16:11:08.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:11:08.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528338 2024-03-09T16:11:08.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528338 2024-03-09T16:11:08.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441 5-128849018917 6-150323855388 7-171798691867 8-193273528338' 2024-03-09T16:11:08.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:08.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:11:08.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364808 2024-03-09T16:11:08.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364808 2024-03-09T16:11:08.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509494 3-85899345972 4-107374182441 5-128849018917 6-150323855388 7-171798691867 8-193273528338 9-214748364808' 2024-03-09T16:11:08.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:08.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-03-09T16:11:08.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:08.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:11:08.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-03-09T16:11:08.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:08.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-03-09T16:11:08.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-03-09T16:11:08.942 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836547 2024-03-09T16:11:08.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:09.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-03-09T16:11:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:09.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-03-09T16:11:09.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:09.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:11:09.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-03-09T16:11:09.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-03-09T16:11:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-03-09T16:11:09.217 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949673022 2024-03-09T16:11:09.218 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:11:09.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-03-09T16:11:09.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:09.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-03-09T16:11:09.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:09.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:11:09.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-03-09T16:11:09.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:09.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-03-09T16:11:09.494 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509494 2024-03-09T16:11:09.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-03-09T16:11:09.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:11:09.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509494 -lt 64424509494 2024-03-09T16:11:09.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:09.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345972 2024-03-09T16:11:09.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:09.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:11:09.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345972 2024-03-09T16:11:09.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:09.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345972 2024-03-09T16:11:09.768 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 85899345972 2024-03-09T16:11:09.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345972' 2024-03-09T16:11:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:10.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345972 -lt 85899345972 2024-03-09T16:11:10.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:10.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182441 2024-03-09T16:11:10.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:10.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:11:10.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182441 2024-03-09T16:11:10.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:10.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182441 2024-03-09T16:11:10.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2024-03-09T16:11:10.045 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.4 seq 107374182441 2024-03-09T16:11:10.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:10.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182441 2024-03-09T16:11:10.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:10.318 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018917 2024-03-09T16:11:10.319 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:10.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:11:10.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018917 2024-03-09T16:11:10.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:10.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018917 2024-03-09T16:11:10.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2024-03-09T16:11:10.321 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.5 seq 128849018917 2024-03-09T16:11:10.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:11:10.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018917 -lt 128849018917 2024-03-09T16:11:10.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:10.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-03-09T16:11:10.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:10.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:11:10.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-03-09T16:11:10.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:10.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-03-09T16:11:10.615 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.6 seq 150323855388 2024-03-09T16:11:10.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-03-09T16:11:10.616 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:11:10.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855388 2024-03-09T16:11:10.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:10.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691867 2024-03-09T16:11:10.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:10.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:11:10.890 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691867 2024-03-09T16:11:10.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:10.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-03-09T16:11:10.892 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.7 seq 171798691867 2024-03-09T16:11:10.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-03-09T16:11:10.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:11:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691868 -lt 171798691867 2024-03-09T16:11:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:11.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528338 2024-03-09T16:11:11.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:11.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:11:11.168 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528338 2024-03-09T16:11:11.168 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528338 2024-03-09T16:11:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2024-03-09T16:11:11.169 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.8 seq 193273528338 2024-03-09T16:11:11.169 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:11:11.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528339 -lt 193273528338 2024-03-09T16:11:11.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:11.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364808 2024-03-09T16:11:11.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:11.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:11:11.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364808 2024-03-09T16:11:11.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:11.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364808 2024-03-09T16:11:11.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-03-09T16:11:11.450 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.9 seq 214748364808 2024-03-09T16:11:11.451 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:11:11.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364809 -lt 214748364808 2024-03-09T16:11:11.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:11:11.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:11.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:12.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:11:12.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:11:12.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:11:12.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:11:12.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:11:12.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:11:12.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:11:12.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:11:12.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:11:12.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:11:12.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:12.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:12.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:11:12.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:11:12.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:11:12.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:11:12.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:11:12.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:11:13.130 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:11:13.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:11:13.600 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:11:13.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:11:13.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:11:13.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T16:11:14.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T16:11:14.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T16:11:14.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T16:11:14.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:11:14.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:11:14.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T16:11:14.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T16:11:14.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:11:14.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T16:11:14.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T16:11:14.548 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:11:14.548 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:9 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:7' 2024-03-09T16:11:14.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T16:11:14.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:11:14.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:11:14.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:11:14.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:11:14.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:11:14.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:11:14.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:11:15.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:11:15.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:11:15.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:11:15.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:11:15.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:11:15.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:11:15.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:11:15.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:11:16.851 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:11:17.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:11:17.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:11:17.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:11:17.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:11:17.369 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:11:17.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:11:17.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:11:17.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:11:17.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:11:17.371 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:11:17.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:11:17.398 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:17.398+0000 7f7d9560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:17.400 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:17.399+0000 7f7d9560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:17.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:17.401+0000 7f7d9560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:17.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:11:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:11:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:17.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:11:17.920 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:17.920+0000 7f7d9560f640 -1 Falling back to public interface 2024-03-09T16:11:17.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:18.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:11:19.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:20.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:11:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:11:20.359 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:20.358+0000 7f7d9560f640 -1 osd.3 67 log_to_monitors true 2024-03-09T16:11:20.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:21.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:11:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/3011438056,v1:127.0.0.1:6827/3011438056] [v2:127.0.0.1:6828/3011438056,v1:127.0.0.1:6829/3011438056] exists,up fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:11:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:11:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:11:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:11:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:11:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:11:21.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:11:21.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:11:21.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:11:21.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:11:21.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:11:21.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:11:21.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:11:21.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:11:21.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:11:22.337 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:11:22.338 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:11:22.338 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:11:22.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:11:22.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:22.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:11:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-03-09T16:11:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-03-09T16:11:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-03-09T16:11:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:22.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:11:22.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673026 2024-03-09T16:11:22.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673026 2024-03-09T16:11:22.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026' 2024-03-09T16:11:22.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:22.734 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:11:22.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-03-09T16:11:22.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-03-09T16:11:22.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498' 2024-03-09T16:11:22.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:22.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:11:22.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-03-09T16:11:22.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-03-09T16:11:22.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018' 2024-03-09T16:11:22.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:22.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:11:23.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182445 2024-03-09T16:11:23.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182445 2024-03-09T16:11:23.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445' 2024-03-09T16:11:23.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:23.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:11:23.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018920 2024-03-09T16:11:23.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018920 2024-03-09T16:11:23.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445 5-128849018920' 2024-03-09T16:11:23.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:23.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:11:23.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855392 2024-03-09T16:11:23.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855392 2024-03-09T16:11:23.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445 5-128849018920 6-150323855392' 2024-03-09T16:11:23.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:23.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:11:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691871 2024-03-09T16:11:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691871 2024-03-09T16:11:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445 5-128849018920 6-150323855392 7-171798691871' 2024-03-09T16:11:23.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:23.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:11:23.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-03-09T16:11:23.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-03-09T16:11:23.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445 5-128849018920 6-150323855392 7-171798691871 8-193273528342' 2024-03-09T16:11:23.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:23.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673026 2-64424509498 3-304942678018 4-107374182445 5-128849018920 6-150323855392 7-171798691871 8-193273528342 9-214748364812' 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-03-09T16:11:23.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:23.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:11:23.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-03-09T16:11:23.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:23.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-03-09T16:11:23.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-03-09T16:11:23.569 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836551 2024-03-09T16:11:23.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:23.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-03-09T16:11:23.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:23.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673026 2024-03-09T16:11:23.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:23.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:11:23.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673026 2024-03-09T16:11:23.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:23.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673026 2024-03-09T16:11:23.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-03-09T16:11:23.843 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673026 2024-03-09T16:11:23.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:11:24.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673026 2024-03-09T16:11:24.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:24.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-03-09T16:11:24.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:24.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509498 2024-03-09T16:11:24.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:11:24.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-03-09T16:11:24.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:24.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-03-09T16:11:24.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:24.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:11:24.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-03-09T16:11:24.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:24.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-03-09T16:11:24.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-03-09T16:11:24.421 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 304942678018 2024-03-09T16:11:24.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:24.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 304942678018 2024-03-09T16:11:24.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:11:25.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:11:25.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:25.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-03-09T16:11:25.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:25.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182445 2024-03-09T16:11:25.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:25.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:11:25.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182445 2024-03-09T16:11:25.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:25.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182445 2024-03-09T16:11:25.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-03-09T16:11:25.972 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 107374182445 2024-03-09T16:11:25.972 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182445 2024-03-09T16:11:26.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:26.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018920 2024-03-09T16:11:26.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:26.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:11:26.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018920 2024-03-09T16:11:26.245 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:26.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018920 2024-03-09T16:11:26.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-03-09T16:11:26.246 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 128849018920 2024-03-09T16:11:26.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:11:26.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018921 -lt 128849018920 2024-03-09T16:11:26.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:26.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855392 2024-03-09T16:11:26.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:26.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:11:26.518 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855392 2024-03-09T16:11:26.518 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:26.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855392 2024-03-09T16:11:26.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-03-09T16:11:26.520 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 150323855392 2024-03-09T16:11:26.520 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:11:26.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855392 -lt 150323855392 2024-03-09T16:11:26.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:26.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691871 2024-03-09T16:11:26.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:26.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:11:26.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691871 2024-03-09T16:11:26.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:26.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691871 2024-03-09T16:11:26.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-03-09T16:11:26.792 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 171798691871 2024-03-09T16:11:26.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:11:27.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691872 -lt 171798691871 2024-03-09T16:11:27.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:27.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-03-09T16:11:27.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:27.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:11:27.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-03-09T16:11:27.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:27.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-03-09T16:11:27.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-03-09T16:11:27.071 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 193273528342 2024-03-09T16:11:27.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:11:27.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528343 -lt 193273528342 2024-03-09T16:11:27.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:27.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-03-09T16:11:27.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:27.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:11:27.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-03-09T16:11:27.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:27.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-03-09T16:11:27.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-03-09T16:11:27.351 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 214748364812 2024-03-09T16:11:27.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:11:27.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364813 -lt 214748364812 2024-03-09T16:11:27.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:11:27.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:27.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:27.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:11:27.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:11:27.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:11:27.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:11:27.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:11:27.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:11:27.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:11:27.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:11:28.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:11:28.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:11:28.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:28.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:28.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:11:28.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:11:28.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:11:28.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:11:28.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:11:28.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:11:28.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:11:28.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:28.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:28.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:28.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:29.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:11:06.036586+0000 2024-03-09T16:11:29.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:11:29.388 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:11:06.036586+0000 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:11:06.036586+0000 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:11:29.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:11:29.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:29.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:29.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:29.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:29.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:06.036586+0000 '>' 2024-03-09T16:11:06.036586+0000 2024-03-09T16:11:29.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:11:30.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:11:30.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:11:30.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:11:30.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:30.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:30.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:30.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:30.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:30.342739+0000 '>' 2024-03-09T16:11:06.036586+0000 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:11:30.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:11:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:11:30.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:11:30.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:11:31.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:11:31.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:11:31.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:11:33.269 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:11:33.270 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:11:33.270 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:11:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:11:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:11:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:11:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:11:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:11:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:11:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:11:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:11:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.9 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:11:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:11:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:33.570+0000 7f3952e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:33.572+0000 7f3952e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:33.573 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:33.573+0000 7f3952e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:33.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:11:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:11:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:11:34.090 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:34.090+0000 7f3952e22640 -1 Falling back to public interface 2024-03-09T16:11:34.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:35.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:35.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:35.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:11:35.485 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:35.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:35.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:11:35.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:36.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:36.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:11:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:11:36.465 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:36.465+0000 7f3952e22640 -1 osd.9 72 log_to_monitors true 2024-03-09T16:11:36.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:37.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:11:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:37.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:11:37.973 INFO:tasks.workunit.client.0.smithi146.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/394564857,v1:127.0.0.1:6875/394564857] [v2:127.0.0.1:6876/394564857,v1:127.0.0.1:6877/394564857] exists,up 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:11:37.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:11:37.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:11:37.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:11:37.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:11:38.058 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:11:38.471 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:11:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:11:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:11:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:11:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-03-09T16:11:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-03-09T16:11:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-03-09T16:11:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:11:38.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-03-09T16:11:38.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-03-09T16:11:38.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031' 2024-03-09T16:11:38.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.680 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:11:38.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509502 2024-03-09T16:11:38.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509502 2024-03-09T16:11:38.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502' 2024-03-09T16:11:38.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:11:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678022 2024-03-09T16:11:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678022 2024-03-09T16:11:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022' 2024-03-09T16:11:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:11:38.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182450 2024-03-09T16:11:38.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182450 2024-03-09T16:11:38.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450' 2024-03-09T16:11:38.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:38.990 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:11:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-03-09T16:11:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-03-09T16:11:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450 5-128849018925' 2024-03-09T16:11:39.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:39.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:11:39.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855396 2024-03-09T16:11:39.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855396 2024-03-09T16:11:39.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450 5-128849018925 6-150323855396' 2024-03-09T16:11:39.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:39.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:11:39.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691875 2024-03-09T16:11:39.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691875 2024-03-09T16:11:39.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450 5-128849018925 6-150323855396 7-171798691875' 2024-03-09T16:11:39.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:39.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:11:39.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528346 2024-03-09T16:11:39.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528346 2024-03-09T16:11:39.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450 5-128849018925 6-150323855396 7-171798691875 8-193273528346' 2024-03-09T16:11:39.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:39.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:11:39.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T16:11:39.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T16:11:39.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509502 3-304942678022 4-107374182450 5-128849018925 6-150323855396 7-171798691875 8-193273528346 9-330712481794' 2024-03-09T16:11:39.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:39.510 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-03-09T16:11:39.510 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836555 2024-03-09T16:11:39.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:39.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836555 2024-03-09T16:11:39.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:39.783 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-03-09T16:11:39.783 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:39.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:11:39.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-03-09T16:11:39.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:39.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-03-09T16:11:39.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-03-09T16:11:39.786 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673031 2024-03-09T16:11:39.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:11:40.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-03-09T16:11:40.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:40.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509502 2024-03-09T16:11:40.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:40.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:11:40.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509502 2024-03-09T16:11:40.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:40.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509502 2024-03-09T16:11:40.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-03-09T16:11:40.094 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509502 2024-03-09T16:11:40.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:11:40.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509502 2024-03-09T16:11:40.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:40.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678022 2024-03-09T16:11:40.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:40.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:11:40.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678022 2024-03-09T16:11:40.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:40.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678022 2024-03-09T16:11:40.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678022' 2024-03-09T16:11:40.368 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 304942678022 2024-03-09T16:11:40.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:40.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678022 -lt 304942678022 2024-03-09T16:11:40.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:40.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182450 2024-03-09T16:11:40.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:40.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:11:40.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182450 2024-03-09T16:11:40.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:40.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182450 2024-03-09T16:11:40.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-03-09T16:11:40.677 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 107374182450 2024-03-09T16:11:40.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:40.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182450 2024-03-09T16:11:40.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:11:41.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:11:41.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:42.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182450 -lt 107374182450 2024-03-09T16:11:42.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:42.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-03-09T16:11:42.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:42.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:11:42.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-03-09T16:11:42.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:42.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-03-09T16:11:42.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-03-09T16:11:42.239 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 128849018925 2024-03-09T16:11:42.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:11:42.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018925 2024-03-09T16:11:42.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:42.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855396 2024-03-09T16:11:42.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:42.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:11:42.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855396 2024-03-09T16:11:42.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:42.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855396 2024-03-09T16:11:42.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-03-09T16:11:42.515 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 150323855396 2024-03-09T16:11:42.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:11:42.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855396 -lt 150323855396 2024-03-09T16:11:42.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:42.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691875 2024-03-09T16:11:42.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691875 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691875 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-03-09T16:11:42.790 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 171798691875 2024-03-09T16:11:42.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:11:43.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691876 -lt 171798691875 2024-03-09T16:11:43.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:43.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528346 2024-03-09T16:11:43.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:43.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:11:43.066 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528346 2024-03-09T16:11:43.066 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:43.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528346 2024-03-09T16:11:43.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-03-09T16:11:43.068 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 193273528346 2024-03-09T16:11:43.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:11:43.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528346 2024-03-09T16:11:43.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:43.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-03-09T16:11:43.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:43.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:11:43.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-03-09T16:11:43.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:43.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T16:11:43.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-03-09T16:11:43.360 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 330712481794 2024-03-09T16:11:43.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:11:43.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-09T16:11:43.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:11:43.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:43.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:11:43.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:11:44.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:11:44.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:11:44.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:44.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:44.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:11:44.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:11:44.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:11:44.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:11:44.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:11:44.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:11:44.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:11:44.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:11:44.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:11:46.754 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:11:47.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:11:47.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:11:47.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.5 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:11:47.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:11:47.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:11:47.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:11:47.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:11:47.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:11:47.300 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:47.300+0000 7f2286c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:47.302 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:47.302+0000 7f2286c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:47.303 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:47.303+0000 7f2286c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:47.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:11:47.555 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:11:47.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:47.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:11:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:47.820+0000 7f2286c13640 -1 Falling back to public interface 2024-03-09T16:11:47.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:11:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:48.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:48.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:11:49.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:50.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:11:50.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:11:50.367+0000 7f2286c13640 -1 osd.5 78 log_to_monitors true 2024-03-09T16:11:50.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:11:51.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:11:51.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:11:51.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:11:51.407 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:51.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:11:51.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:11:51.891 INFO:tasks.workunit.client.0.smithi146.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/2742134980,v1:127.0.0.1:6843/2742134980] [v2:127.0.0.1:6844/2742134980,v1:127.0.0.1:6845/2742134980] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:11:51.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:11:51.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:11:51.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:11:51.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:11:51.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:11:51.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:11:51.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:11:51.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:11:52.352 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:11:52.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:11:52.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:52.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:11:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-03-09T16:11:52.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-03-09T16:11:52.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-03-09T16:11:52.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:52.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:11:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-03-09T16:11:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-03-09T16:11:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034' 2024-03-09T16:11:52.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:52.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:11:52.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-03-09T16:11:52.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-03-09T16:11:52.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506' 2024-03-09T16:11:52.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:52.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:11:52.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-03-09T16:11:52.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-03-09T16:11:52.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026' 2024-03-09T16:11:52.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:52.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:11:53.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182453 2024-03-09T16:11:53.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182453 2024-03-09T16:11:53.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453' 2024-03-09T16:11:53.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:53.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:11:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-09T16:11:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-09T16:11:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453 5-352187318274' 2024-03-09T16:11:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:53.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:11:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855400 2024-03-09T16:11:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855400 2024-03-09T16:11:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453 5-352187318274 6-150323855400' 2024-03-09T16:11:53.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:53.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:11:53.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-03-09T16:11:53.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-03-09T16:11:53.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453 5-352187318274 6-150323855400 7-171798691879' 2024-03-09T16:11:53.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:53.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:11:53.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-03-09T16:11:53.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-03-09T16:11:53.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453 5-352187318274 6-150323855400 7-171798691879 8-193273528350' 2024-03-09T16:11:53.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:11:53.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:11:53.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-03-09T16:11:53.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-03-09T16:11:53.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673034 2-64424509506 3-304942678026 4-107374182453 5-352187318274 6-150323855400 7-171798691879 8-193273528350 9-330712481798' 2024-03-09T16:11:53.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:53.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-03-09T16:11:53.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:53.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:11:53.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-03-09T16:11:53.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:53.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-03-09T16:11:53.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-03-09T16:11:53.601 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836559 2024-03-09T16:11:53.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:11:53.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-03-09T16:11:53.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:53.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-03-09T16:11:53.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:53.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:11:53.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-03-09T16:11:53.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:53.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-03-09T16:11:53.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-03-09T16:11:53.879 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673034 2024-03-09T16:11:53.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:11:54.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673034 2024-03-09T16:11:54.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:54.180 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-03-09T16:11:54.180 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:54.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:11:54.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-03-09T16:11:54.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:54.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-03-09T16:11:54.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-03-09T16:11:54.183 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509506 2024-03-09T16:11:54.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:11:54.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-03-09T16:11:54.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:54.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-03-09T16:11:54.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:54.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:11:54.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-03-09T16:11:54.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:54.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-03-09T16:11:54.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-03-09T16:11:54.460 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 304942678026 2024-03-09T16:11:54.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:11:54.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678026 -lt 304942678026 2024-03-09T16:11:54.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:54.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182453 2024-03-09T16:11:54.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:54.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:11:54.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182453 2024-03-09T16:11:54.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:54.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182453 2024-03-09T16:11:54.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182453' 2024-03-09T16:11:54.779 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 107374182453 2024-03-09T16:11:54.779 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:11:55.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182453 2024-03-09T16:11:55.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:55.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318274 2024-03-09T16:11:55.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:55.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:11:55.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318274 2024-03-09T16:11:55.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:55.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-09T16:11:55.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-03-09T16:11:55.058 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 352187318274 2024-03-09T16:11:55.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:11:55.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-09T16:11:55.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:55.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855400 2024-03-09T16:11:55.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:55.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:11:55.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855400 2024-03-09T16:11:55.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:55.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855400 2024-03-09T16:11:55.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-03-09T16:11:55.338 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 150323855400 2024-03-09T16:11:55.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:11:55.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855400 -lt 150323855400 2024-03-09T16:11:55.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:55.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-03-09T16:11:55.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:55.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:11:55.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-03-09T16:11:55.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-03-09T16:11:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-03-09T16:11:55.614 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 171798691879 2024-03-09T16:11:55.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:11:55.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-03-09T16:11:55.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:55.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-03-09T16:11:55.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:55.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:11:55.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-03-09T16:11:55.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-03-09T16:11:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-03-09T16:11:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 193273528350 2024-03-09T16:11:55.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:11:56.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528350 2024-03-09T16:11:56.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:11:56.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-03-09T16:11:56.162 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:11:56.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:11:56.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-03-09T16:11:56.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:11:56.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-03-09T16:11:56.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-03-09T16:11:56.164 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 330712481798 2024-03-09T16:11:56.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:11:56.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-03-09T16:11:56.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:11:56.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:56.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:56.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:11:56.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:11:57.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:11:57.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:11:57.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:11:57.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:11:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:11:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:11:57.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:11:57.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:11:57.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:11:57.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:11:57.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:11:57.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:11:57.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:57.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:57.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:11:30.342739+0000 2024-03-09T16:11:57.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:11:58.237 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:11:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:11:30.342739+0000 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:11:30.342739+0000 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:58.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:30.342739+0000 '>' 2024-03-09T16:11:30.342739+0000 2024-03-09T16:11:58.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:11:59.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:11:59.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:11:59.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:11:59.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:11:59.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:11:59.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:11:59.519 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:11:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:30.342739+0000 '>' 2024-03-09T16:11:30.342739+0000 2024-03-09T16:11:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:12:00.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:12:00.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:12:00.798 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:12:00.798 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:12:00.798 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:12:00.798 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:12:00.798 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:12:01.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:59.171531+0000 '>' 2024-03-09T16:11:30.342739+0000 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:12:01.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:01.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:01.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:01.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:01.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:01.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:01.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:12:01.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:01.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:01.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:12:03.111 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:12:03.111 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:12:03.111 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:03.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:03.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:12:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:12:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.9 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:03.408 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:12:03.408+0000 7efd6440e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:03.410 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:12:03.410+0000 7efd6440e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:03.412 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:12:03.411+0000 7efd6440e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:12:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:03.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:03.930 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:12:03.930+0000 7efd6440e640 -1 Falling back to public interface 2024-03-09T16:12:03.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:05.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:06.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:06.273 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:12:06.272+0000 7efd6440e640 -1 osd.9 84 log_to_monitors true 2024-03-09T16:12:06.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:07.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:07.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:07.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:07.512 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:12:07.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:07.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/1208618573,v1:127.0.0.1:6875/1208618573] [v2:127.0.0.1:6876/1208618573,v1:127.0.0.1:6877/1208618573] exists,up 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:07.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:07.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:07.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:07.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:08.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:08.229 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:12:08.229 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:08.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-03-09T16:12:08.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-03-09T16:12:08.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-03-09T16:12:08.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:08.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-03-09T16:12:08.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-03-09T16:12:08.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039' 2024-03-09T16:12:08.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-03-09T16:12:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-03-09T16:12:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510' 2024-03-09T16:12:08.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678030 2024-03-09T16:12:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678030 2024-03-09T16:12:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030' 2024-03-09T16:12:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.827 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:08.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-03-09T16:12:08.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-03-09T16:12:08.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458' 2024-03-09T16:12:08.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:08.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:09.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318278 2024-03-09T16:12:09.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318278 2024-03-09T16:12:09.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458 5-352187318278' 2024-03-09T16:12:09.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:09.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:09.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-03-09T16:12:09.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-03-09T16:12:09.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458 5-352187318278 6-150323855404' 2024-03-09T16:12:09.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:09.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:09.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-03-09T16:12:09.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-03-09T16:12:09.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458 5-352187318278 6-150323855404 7-171798691883' 2024-03-09T16:12:09.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:09.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:09.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-03-09T16:12:09.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-03-09T16:12:09.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458 5-352187318278 6-150323855404 7-171798691883 8-193273528354' 2024-03-09T16:12:09.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:09.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:09.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-09T16:12:09.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-09T16:12:09.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509510 3-304942678030 4-107374182458 5-352187318278 6-150323855404 7-171798691883 8-193273528354 9-377957122050' 2024-03-09T16:12:09.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:09.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-03-09T16:12:09.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:09.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:09.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-03-09T16:12:09.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:09.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-03-09T16:12:09.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-03-09T16:12:09.495 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836563 2024-03-09T16:12:09.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:09.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-03-09T16:12:09.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:09.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-03-09T16:12:09.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:09.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-03-09T16:12:09.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:09.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-03-09T16:12:09.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-03-09T16:12:09.771 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673039 2024-03-09T16:12:09.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:10.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673039 2024-03-09T16:12:10.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:10.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-03-09T16:12:10.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:10.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:10.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-03-09T16:12:10.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:10.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-03-09T16:12:10.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-03-09T16:12:10.048 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509510 2024-03-09T16:12:10.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:10.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-03-09T16:12:10.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:10.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678030 2024-03-09T16:12:10.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:10.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:10.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678030 2024-03-09T16:12:10.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:10.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678030 2024-03-09T16:12:10.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678030' 2024-03-09T16:12:10.327 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 304942678030 2024-03-09T16:12:10.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:10.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678030 -lt 304942678030 2024-03-09T16:12:10.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:10.602 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-03-09T16:12:10.602 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:10.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:10.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-03-09T16:12:10.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:10.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-03-09T16:12:10.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-03-09T16:12:10.605 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 107374182458 2024-03-09T16:12:10.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:10.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182458 -lt 107374182458 2024-03-09T16:12:10.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:10.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318278 2024-03-09T16:12:10.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:10.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:10.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318278 2024-03-09T16:12:10.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:10.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318278 2024-03-09T16:12:10.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318278' 2024-03-09T16:12:10.881 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 352187318278 2024-03-09T16:12:10.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:11.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318279 -lt 352187318278 2024-03-09T16:12:11.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:11.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-03-09T16:12:11.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:11.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:11.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-03-09T16:12:11.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:11.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-03-09T16:12:11.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-03-09T16:12:11.186 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 150323855404 2024-03-09T16:12:11.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:11.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855404 2024-03-09T16:12:11.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:11.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-03-09T16:12:11.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:11.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:11.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-03-09T16:12:11.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:11.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-03-09T16:12:11.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-03-09T16:12:11.466 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 171798691883 2024-03-09T16:12:11.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:11.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-03-09T16:12:11.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:11.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-03-09T16:12:11.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:11.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:11.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-03-09T16:12:11.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:11.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-03-09T16:12:11.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-03-09T16:12:11.740 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 193273528354 2024-03-09T16:12:11.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:12.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528355 -lt 193273528354 2024-03-09T16:12:12.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:12.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-03-09T16:12:12.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:12.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:12:12.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-03-09T16:12:12.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:12.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-09T16:12:12.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-03-09T16:12:12.033 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 377957122050 2024-03-09T16:12:12.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:12:12.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-09T16:12:12.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:12:12.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:12.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:12.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:12:12.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:12.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:12.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:12.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:12.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:12.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:12:12.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:12.911 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:12.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:13.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:12:13.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:12:13.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:12:13.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:12:13.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:13.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229385"' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229385' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229387"' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229387' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229388: /' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229385 229387' 2024-03-09T16:12:13.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:12:13.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:12:13.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229385 2024-03-09T16:12:13.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:12:13.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229390: /' 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:16.268 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:16.269 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:16.270 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:16.271 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:16.271 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:16.271 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:12:16.271 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:12:16.271 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:16.272 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: start osd.5 2024-03-09T16:12:16.273 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:16.328 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.328 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:16.328 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:16.329 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:16.330 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:16.331 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:16.331 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:16.332 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: start osd.9 2024-03-09T16:12:16.333 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2024-03-09T16:12:16.298+0000 7f61e840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2024-03-09T16:12:16.301+0000 7f61e840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.774 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2024-03-09T16:12:16.302+0000 7f61e840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 0 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2024-03-09T16:12:16.816+0000 7f61e840d640 -1 Falling back to public interface 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.775 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 1 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2024-03-09T16:12:19.500+0000 7f61e840d640 -1 osd.5 89 log_to_monitors true 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.776 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.777 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:20.777 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 3 2024-03-09T16:12:20.777 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.777 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229388: osd.5 up in weight 1 up_from 94 up_thru 82 down_at 90 last_229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2024-03-09T16:12:16.358+0000 7fc9fde97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2024-03-09T16:12:16.361+0000 7fc9fde97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2024-03-09T16:12:16.362+0000 7fc9fde97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:20.791 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 0 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2024-03-09T16:12:16.882+0000 7fc9fde97640 -1 Falling back to public interface 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:20.792 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 1 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2024-03-09T16:12:19.725+0000 7fc9fde97640 -1 osd.9 89 log_to_monitors true 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:20.793 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 3 2024-03-09T16:12:20.794 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:20.794 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: osd.9 up in weight 1 up_from 94 up_thru 0 down_at 90 last_clean_interval [88,89) [v2:127.0.0.1:6874/1334787312,v1:127.0.0.1:6875/1334787312] [v2:127.0.0.1:6876/1334787312,v1:127.0.0.1:6877/1334787312] exists,up 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:21.513 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 1 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 2 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 3 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 4 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 5 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 6 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 7 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 8 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: 9' 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:21.514 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042' 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [82,89) [v2:127.0.0.1:6842/1611397725,v1:127.0.0.1:6843/1611397725] [v2:127.0.0.1:6844/1611397725,v1:127.0.0.1:6845/1611397725] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:12:21.515 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:21.516 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 1 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 2 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 3 2024-03-09T16:12:21.517 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 4 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 5 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 6 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 7 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 8 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: 9' 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:21.518 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:21.519 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-09T16:12:21.519 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-09T16:12:21.519 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043' 2024-03-09T16:12:21.519 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/q/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514' 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678035 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678035 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035' 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:22.800 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462' 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462 5-403726925826' 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-03-09T16:12:22.801 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462 5-403726925826 6-150323855408' 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691888 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462 5-403726925826 6-150323855408 7-171798691888' 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clonea/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509513 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509513 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513' 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678034 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678034 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034' 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182461 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182461 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461' 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461 5-403726925827' 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461 5-403726925827 6-150323855409' 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461 5-403726925827 6-150323855409 7-171798691887' 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:23.525 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clon.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-03-09T16:12:23.525 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462 5-403726925826 6-150323855408 7-171798691888 8-193273528359' 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673042 2-64424509514 3-304942678035 4-107374182462 5-403726925826 6-150323855408 7-171798691888 8-193273528359 9-403726925827' 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:23.526 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.0 seq 21474836566 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836566 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.1 seq 42949673042 2024-03-09T16:12:23.527 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673042 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.ce.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-09T16:12:23.528 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461 5-403726925827 6-150323855409 7-171798691887 8-193273528358' 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509513 3-304942678034 4-107374182461 5-403726925827 6-150323855409 7-171798691887 8-193273528358 9-403726925826' 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-03-09T16:12:23.529 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.0 seq 21474836567 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836567 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.1 seq 42949673043 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:23.530 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509513 2024-03-09T16:12:23.531 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:23.531 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:24.429 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.lient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-03-09T16:12:24.429 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.429 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-03-09T16:12:24.429 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.2 seq 64424509514 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678035 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678035 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678035 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678035' 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.3 seq 304942678035 2024-03-09T16:12:24.430 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678035 -lt 304942678035 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.4 seq 107374182462 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182462 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925826 2024-03-09T16:12:24.431 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.432 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:24.432 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925826 2024-03-09T16:12:24.432 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.432 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiticlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509513 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509513 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.2 seq 64424509513 2024-03-09T16:12:24.474 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509513 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678034 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678034 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678034 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678034' 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.3 seq 304942678034 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678035 -lt 304942678034 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.475 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182461 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182461 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182461 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.4 seq 107374182461 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182461 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925827 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:24.476 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925827 2024-03-09T16:12:24.477 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:24.477 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 403726925827' 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.5 seq 403726925827 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.683 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.6 seq 150323855409 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855409 -lt 150323855409 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-03-09T16:12:25.684 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.7 seq 171798691887 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691888 -lt 171798691887 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.8 seq 193273528358 2024-03-09T16:12:25.685 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:25.686 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 403726925826' 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.5 seq 403726925826 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:25.689 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.6 seq 150323855408 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855409 -lt 150323855408 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-03-09T16:12:25.690 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.7 seq 171798691888 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691888 -lt 171798691888 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.8 seq 193273528359 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528359 2024-03-09T16:12:27.108 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgg_stats: for s in $seqs 2024-03-09T16:12:27.108 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925826 2024-03-09T16:12:27.108 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:27.108 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925826 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925826' 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: waiting osd.9 seq 403726925826 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:12:27.109 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:12:27.110 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:12:27.111 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:12:27.111 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:12:27.111 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:12:27.111 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:27.111 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ub_stats: for s in $seqs 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925827 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925827 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:27.201 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925827' 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: waiting osd.9 seq 403726925827 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:27.202 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:27.203 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:27.750 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:229388: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:12:27.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:12:27.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229387 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:12:27.832 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:229390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:12:27.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:12:28.110 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:12:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:12:28.637 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:12:28.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:12:28.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:12:28.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:12:28.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:12:28.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:12:28.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:12:28.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:59.171531+0000 '>' 2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:28.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:12:29.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:12:29.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:12:29.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:12:29.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:12:29.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:12:29.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:12:29.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:12:30.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:11:59.171531+0000 '>' 2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:30.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:12:31.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:12:31.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:12:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:12:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:12:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:12:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:12:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:12:29.359099+0000 '>' 2024-03-09T16:11:59.171531+0000 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232839"' 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232839' 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:31.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232841"' 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232841' 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 232839 232841' 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232839 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232842: /' 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232844: /' 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:34.823 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:34.824 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: _ 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: hinfo_key 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: snapset 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:34.825 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:34.826 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:34.827 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:12:34.828 INFO:tasks.workunit.client.0.smithi146.stderr:232844: start osd.9 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:12:34.953 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.954 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: _ 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: hinfo_key 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: snapset 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:34.955 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:34.956 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:34.957 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:12:34.958 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:12:34.958 INFO:tasks.workunit.client.0.smithi146.stderr:232842: start osd.5 2024-03-09T16:12:39.442 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:39.442 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:12:39.442 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2024-03-09T16:12:34.855+0000 7ff92ed06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2024-03-09T16:12:34.857+0000 7ff92ed06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2024-03-09T16:12:34.859+0000 7ff92ed06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:39.443 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 0 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2024-03-09T16:12:35.379+0000 7ff92ed06640 -1 Falling back to public interface 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 1 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.444 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2024-03-09T16:12:37.811+0000 7ff92ed06640 -1 osd.9 96 log_to_monitors true 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 3 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.445 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232844: osd.9 up in weight 1 up_from 101 up_thru 0 down_aovery_ops 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2024-03-09T16:12:34.982+0000 7f666aaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2024-03-09T16:12:34.984+0000 7f666aaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2024-03-09T16:12:34.985+0000 7f666aaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:12:39.889 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 0 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2024-03-09T16:12:35.504+0000 7f666aaa2640 -1 Falling back to public interface 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 1 2024-03-09T16:12:39.890 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2024-03-09T16:12:38.177+0000 7f666aaa2640 -1 osd.5 96 log_to_monitors true 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 3 2024-03-09T16:12:39.891 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:39.892 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ct 97 last_clean_interval [94,96) [v2:127.0.0.1:6842/246583175,v1:127.0.0.1:6843/246583175] [v2:127.0.0.1:6844/246583175,v1:127.0.0.1:6845/246583175] exists,up 74620015-395d-484e-afe7-a1d35ddc2ca4 2024-03-09T16:12:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:40.197 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:40.198 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 1 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 2 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 3 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 4 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 5 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 6 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 7 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 8 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: 9' 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:40.199 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048' 2024-03-09T16:12:40.200 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.721 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:41.721 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-03-09T16:12:41.721 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-03-09T16:12:41.721 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519' 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678040 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678040 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040' 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467' 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.722 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467 5-446676598786' 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467 5-446676598786 6-150323855413' 2024-03-09T16:12:41.723 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691893 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467 5-446676598786 6-150323855413 7-171798691893' 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:41.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:42.207 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephteseph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:42.207 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:42.207 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:42.207 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 4 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: osd.5 up in weight 1 up_from 104 up_thru 104 down_at 97 last_clean_interval [94,96) [v2:127.0.0.1:6874/2210658086,v1:127.0.0.1:6875/2210658086] [v2:127.0.0.1:6876/2210658086,v1:127.0.0.1:6877/2210658086] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:42.208 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 1 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 2 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 3 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 4 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 5 2024-03-09T16:12:42.209 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 6 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 7 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 8 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: 9' 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-03-09T16:12:42.210 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $t/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467 5-446676598786 6-150323855413 7-171798691893 8-193273528364' 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-03-09T16:12:42.725 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673048 2-64424509519 3-304942678040 4-107374182467 5-446676598786 6-150323855413 7-171798691893 8-193273528364 9-433791696898' 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.0 seq 21474836572 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-03-09T16:12:42.726 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.1 seq 42949673048 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:42.727 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:43.004 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/ids 2024-03-09T16:12:43.004 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:43.004 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-03-09T16:12:43.004 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049' 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521' 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678041 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678041 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041' 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:43.005 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468' 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468 5-446676598787' 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468 5-446676598787 6-150323855415' 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468 5-446676598787 6-150323855415 7-171798691894' 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-03-09T16:12:43.547 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468 5-446676598787 6-150323855415 7-171798691894 8-193273528365' 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696900 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696900 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673049 2-64424509521 3-304942678041 4-107374182468 5-446676598787 6-150323855415 7-171798691894 8-193273528365 9-433791696900' 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.548 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.0 seq 21474836574 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-03-09T16:12:43.549 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.1 seq 42949673049 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.2 seq 64424509519 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:43.693 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678040 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678040 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678040 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678040' 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.3 seq 304942678040 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678041 -lt 304942678040 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:43.694 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.4 seq 107374182467 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182467 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-03-09T16:12:43.695 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:43.696 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo4: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673049 -lt 42949673049 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.2 seq 64424509521 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509521 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678041 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678041 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.805 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678041 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678041' 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.3 seq 304942678041 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678041 -lt 304942678041 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-03-09T16:12:44.806 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-03-09T16:12:44.807 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.4 seq 107374182468 2024-03-09T16:12:44.807 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:44.807 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182468 2024-03-09T16:12:44.807 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.807 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-03-09T16:12:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //h 'waiting osd.5 seq 446676598786' 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.5 seq 446676598786 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598786 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.6 seq 150323855413 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855413 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.7 seq 171798691893 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691893 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:44.918 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.8 seq 193273528364 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:44.919 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528364 -lt 193273528364 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.5 seq 446676598787 2024-03-09T16:12:45.771 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.6 seq 150323855415 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855415 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:45.772 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.7 seq 171798691894 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691894 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:45.773 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-03-09T16:12:46.265 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flushlush_pg_stats: for s in $seqs 2024-03-09T16:12:46.265 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696898 2024-03-09T16:12:46.265 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:46.265 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:12:46.265 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696898 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696898' 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: waiting osd.9 seq 433791696898 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696898 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:46.266 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:232844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:12:46.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:12:47.327 INFO:tasks.workunit.client.0.smithi146.stderr:_pg_stats: cut -d - -f 2 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.8 seq 193273528365 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528365 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696900 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696900 2024-03-09T16:12:47.328 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696900 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696900' 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: waiting osd.9 seq 433791696900 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696900 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:12:47.329 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:232842: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232841 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:12:47.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:12:47.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:12:47.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:12:47.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:12:47.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:12:47.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:12:47.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-09T16:12:47.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:12:47.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:12:47.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T16:12:47.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236098"' 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236098' 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236100"' 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236100' 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236101: /' 2024-03-09T16:12:47.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 236098 236100' 2024-03-09T16:12:47.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:12:47.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:12:47.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236098 2024-03-09T16:12:47.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:47.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236103: /' 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:12:50.204 INFO:tasks.workunit.client.0.smithi146.stderr:236103: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:50.205 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:50.206 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:50.206 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:50.206 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:50.207 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: start osd.5 2024-03-09T16:12:50.208 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:12:50.308 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:12:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:12:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:12:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: start osd.3 2024-03-09T16:12:50.312 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2024-03-09T16:12:50.235+0000 7ff32860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2024-03-09T16:12:50.238+0000 7ff32860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2024-03-09T16:12:50.239+0000 7ff32860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:12:54.915 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 0 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2024-03-09T16:12:50.754+0000 7ff32860f640 -1 Falling back to public interface 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 1 2024-03-09T16:12:54.916 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2024-03-09T16:12:53.433+0000 7ff32860f640 -1 osd.5 106 log_to_monitors true 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 3 2024-03-09T16:12:54.917 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236103: osd.5 up in weight 1 up_from 112 up_thru 104 down_at 107 l236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2024-03-09T16:12:50.337+0000 7f659aa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.918 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2024-03-09T16:12:50.339+0000 7f659aa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2024-03-09T16:12:50.340+0000 7f659aa0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 0 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2024-03-09T16:12:50.856+0000 7f659aa0e640 -1 Falling back to public interface 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.919 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 1 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:12:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2024-03-09T16:12:53.583+0000 7f659aa0e640 -1 osd.3 106 log_to_monitors true 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 3 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:12:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:12:55.994 INFO:tasks.workunit.client.0.smithi146.stderr:236101: osd.3 up in weight 1 up_from 113 up_thru 104 down_at 107 last_clean_interval [104,106) [v2:127.0.0.1:6826/3823210309,v1:127.0.0.1:6827/3823210309] [v2:127.0.0.1:6828/3823210309,v1:127.0.0.1:6829/3823210309] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:55.995 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 1 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 2 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 3 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 4 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 5 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 6 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 7 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 8 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: 9' 2024-03-09T16:12:55.996 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053' 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.clieast_clean_interval [71,106) [v2:127.0.0.1:6874/4085218221,v1:127.0.0.1:6875/4085218221] [v2:127.0.0.1:6876/4085218221,v1:127.0.0.1:6877/4085218221] exists,up fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:12:55.997 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:12:55.998 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 1 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 2 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 3 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 4 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 5 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 6 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 7 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 8 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: 9' 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:12:55.999 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054' 2024-03-09T16:12:56.000 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525' 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-03-09T16:12:57.110 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450' 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473' 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337155 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337155 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473 5-481036337155' 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-03-09T16:12:57.111 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473 5-481036337155 6-150323855418' 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473 5-481036337155 6-150323855418 7-171798691898' 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524' 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-03-09T16:12:57.112 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451' 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472' 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472 5-481036337154' 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-03-09T16:12:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472 5-481036337154 6-150323855419' 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472 5-481036337154 6-150323855419 7-171798691899' 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.114 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtes/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473 5-481036337155 6-150323855418 7-171798691898 8-193273528370' 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696903 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696903 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509525 3-485331304450 4-107374182473 5-481036337155 6-150323855418 7-171798691898 8-193273528370 9-433791696903' 2024-03-09T16:12:57.792 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.0 seq 21474836577 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836577 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.793 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.1 seq 42949673054 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.794 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/ct/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472 5-481036337154 6-150323855419 7-171798691899 8-193273528369' 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696904 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696904 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509524 3-485331304451 4-107374182472 5-481036337154 6-150323855419 7-171798691899 8-193273528369 9-433791696904' 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-03-09T16:12:57.800 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.0 seq 21474836578 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836578 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.1 seq 42949673053 2024-03-09T16:12:57.801 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:12:57.802 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673053 2024-03-09T16:12:57.802 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:57.802 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-03-09T16:12:57.802 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:57.802 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:12:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-03-09T16:12:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-03-09T16:12:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.2 seq 64424509524 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304451 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304451 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304451' 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.3 seq 485331304451 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-03-09T16:12:58.723 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.4 seq 107374182472 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182472 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337154 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:58.724 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337154 2024-03-09T16:12:58.725 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.725 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echolone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.2 seq 64424509525 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304450 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304450 2024-03-09T16:12:58.771 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304450' 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.3 seq 485331304450 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.4 seq 107374182473 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:12:58.772 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182473 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337155 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337155 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:58.773 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337155 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 481036337154' 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.5 seq 481036337154 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337154 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.6 seq 150323855419 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855419 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:12:59.866 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.7 seq 171798691899 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691899 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-09T16:12:59.867 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-09T16:12:59.868 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.8 seq 193273528369 2024-03-09T16:12:59.868 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:12:59.868 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: f'waiting osd.5 seq 481036337155' 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.5 seq 481036337155 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337155 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-03-09T16:13:00.020 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.6 seq 150323855418 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855418 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:00.021 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.7 seq 171798691898 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691898 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.8 seq 193273528370 2024-03-09T16:13:00.022 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:13:00.023 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528370 2024-03-09T16:13:01.278 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fllush_pg_stats: for s in $seqs 2024-03-09T16:13:01.278 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696904 2024-03-09T16:13:01.278 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:01.278 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696904 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696904 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696904' 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: waiting osd.9 seq 433791696904 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696905 -lt 433791696904 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:13:01.279 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:13:01.280 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:13:01.281 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:13:01.281 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:13:01.281 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //hush_pg_stats: for s in $seqs 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696903 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696903 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696903 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696903' 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: waiting osd.9 seq 433791696903 2024-03-09T16:13:01.390 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696905 -lt 433791696903 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:13:01.391 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:236101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:236101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:13:01.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:13:01.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:13:01.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236100 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:13:01.954 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:236103: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:236103: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:13:01.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:13:02.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:13:02.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:02.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:13:02.776 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:13:02.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:13:02.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:13:02.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:13:02.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:13:02.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:13:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:12:29.359099+0000 '>' 2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:03.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:13:04.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:13:04.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:13:04.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:13:04.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:13:04.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:13:04.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:13:04.061 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:13:04.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:12:29.359099+0000 '>' 2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:04.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:13:05.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:13:05.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:13:05.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:13:05.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:13:05.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:13:05.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:13:05.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:13:03.465667+0000 '>' 2024-03-09T16:12:29.359099+0000 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239503"' 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239503' 2024-03-09T16:13:05.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239505"' 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239505' 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 239503 239505' 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239506: /' 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239503 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:13:05.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239508: /' 2024-03-09T16:13:08.661 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:13:08.661 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:13:08.661 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:13:08.661 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:13:08.661 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:13:08.662 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: _ 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: hinfo_key 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: snapset 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:13:08.663 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:13:08.664 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:13:08.664 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:13:08.664 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:13:08.664 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:13:08.664 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:13:08.666 INFO:tasks.workunit.client.0.smithi146.stderr:239506: start osd.3 2024-03-09T16:13:09.139 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:13:09.140 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: _ 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: hinfo_key 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: snapset 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:13:09.141 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:13:09.142 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:13:09.143 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:13:09.144 INFO:tasks.workunit.client.0.smithi146.stderr:239508: start osd.5 2024-03-09T16:13:12.836 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:13:12.836 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:13:12.836 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:13:12.836 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2024-03-09T16:13:08.693+0000 7f53ee5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2024-03-09T16:13:08.695+0000 7f53ee5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2024-03-09T16:13:08.697+0000 7f53ee5e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:13:12.837 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 0 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2024-03-09T16:13:09.230+0000 7f53ee5e1640 -1 Falling back to public interface 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 1 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:12.838 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2024-03-09T16:13:11.699+0000 7f53ee5e1640 -1 osd.3 115 log_to_monitors true 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 3 2024-03-09T16:13:12.839 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2024-03-09T16:13:09.167+0000 7f1fcd938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:13.984 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2024-03-09T16:13:09.170+0000 7f1fcd938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2024-03-09T16:13:09.171+0000 7f1fcd938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 0 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2024-03-09T16:13:09.685+0000 7f1fcd938640 -1 Falling back to public interface 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:13.985 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 1 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:13.986 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2024-03-09T16:13:12.573+0000 7f1fcd938640 -1 osd.5 115 log_to_monitors true 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 3 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:13.987 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: osd.3 up in weight 1 up_from 120 up_thru 113 down_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6826/1934862355,v1:127.0.0.1:6827/1934862355] [v2:127.0.0.1:6828/1934862355,v1:127.0.0.1:6829/1934862355] exists,up fc1f68e9-3c93-4488-b56c-b09f36aba9ea 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:13:14.000 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 1 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 2 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 3 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 4 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 5 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 6 2024-03-09T16:13:14.001 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 7 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 8 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: 9' 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-03-09T16:13:14.002 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059' 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530' 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-03-09T16:13:15.334 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522' 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478' 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478 5-528280977410' 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-03-09T16:13:15.335 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478 5-528280977410 6-150323855424' 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478 5-528280977410 6-150323855424 7-171798691903' 2024-03-09T16:13:15.336 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 4 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: osd.5 up in weight 1 up_from 123 up_thru 123 down_at 116 last_clean_interval [112,115) [v2:127.0.0.1:6874/916901514,v1:127.0.0.1:6875/916901514] [v2:127.0.0.1:6876/916901514,v1:127.0.0.1:6877/916901514] exists,up ecf7f5db-2c6b-400d-8c88-a4d3c606d8da 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:13:16.084 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:13:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 1 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 2 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 3 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 4 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 5 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 6 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 7 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 8 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: 9' 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:13:16.087 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-09T16:13:16.087 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-09T16:13:16.087 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $a/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478 5-528280977410 6-150323855424 7-171798691903 8-193273528375' 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:13:16.590 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696909 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696909 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673059 2-64424509530 3-515396075522 4-107374182478 5-528280977410 6-150323855424 7-171798691903 8-193273528375 9-433791696909' 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.0 seq 21474836583 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:13:16.591 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836583 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.1 seq 42949673059 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-03-09T16:13:16.592 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cepids 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060' 2024-03-09T16:13:16.896 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531' 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523' 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479' 2024-03-09T16:13:16.897 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977411 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977411 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479 5-528280977411' 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479 5-528280977411 6-150323855425' 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:16.898 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:13:17.457 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691905 2024-03-09T16:13:17.457 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691905 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479 5-528280977411 6-150323855425 7-171798691905' 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479 5-528280977411 6-150323855425 7-171798691905 8-193273528376' 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696910 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696910 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673060 2-64424509531 3-515396075523 4-107374182479 5-528280977411 6-150323855425 7-171798691905 8-193273528376 9-433791696910' 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:17.458 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.0 seq 21474836584 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:13:17.459 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-03-09T16:13:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-03-09T16:13:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-03-09T16:13:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.1 seq 42949673060 2024-03-09T16:13:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221htest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.2 seq 64424509530 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075522 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075522 2024-03-09T16:13:17.532 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075522' 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.3 seq 515396075522 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.4 seq 107374182478 2024-03-09T16:13:17.533 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182478 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977410 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977410 2024-03-09T16:13:17.534 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.686 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=52824: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.2 seq 64424509531 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075523 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.687 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075523 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075523' 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.3 seq 515396075523 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-03-09T16:13:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-03-09T16:13:18.689 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.4 seq 107374182479 2024-03-09T16:13:18.689 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:13:18.689 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182479 2024-03-09T16:13:18.689 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.689 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977411 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //h80977410 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977410' 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.5 seq 528280977410 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977411 -lt 528280977410 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:13:18.735 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.6 seq 150323855424 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855424 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-03-09T16:13:18.736 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.7 seq 171798691903 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691903 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.8 seq 193273528375 2024-03-09T16:13:18.737 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:13:19.555 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:19.555 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:13:19.555 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977411 2024-03-09T16:13:19.555 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977411 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977411' 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.5 seq 528280977411 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977411 -lt 528280977411 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-03-09T16:13:19.556 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.6 seq 150323855425 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855425 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691905 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691905 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691905 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.7 seq 171798691905 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691905 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:19.557 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-03-09T16:13:19.558 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:19.558 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:13:19.558 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-03-09T16:13:20.133 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush93273528375 -lt 193273528375 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696909 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696909 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696909 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696909' 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: waiting osd.9 seq 433791696909 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696909 -lt 433791696909 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:13:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:13:20.136 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:13:20.136 INFO:tasks.workunit.client.0.smithi146.stderr:239506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:13:20.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:13:20.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:13:20.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239505 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:_pg_stats: cut -d - -f 2 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.8 seq 193273528376 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528376 -lt 193273528376 2024-03-09T16:13:21.109 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-433791696910 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-433791696910 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696910 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 433791696910' 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: waiting osd.9 seq 433791696910 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696911 -lt 433791696910 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:21.110 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:13:21.111 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:239508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:13:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:13:21.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:13:21.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:13:21.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:13:21.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:13:21.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:13:21.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:13:21.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:13:21.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:13:21.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:13:21.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:13:21.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:13:21.307 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:13:21.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:13:21.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:13:21.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:13:21.309 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:13:21.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:13:21.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:13:21.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:13:21.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:13:21.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:21.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:21.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:13:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:13:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:13:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:13:21.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:13:21.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:13:21.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:13:21.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:13:21.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:13:21.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:13:21.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:13:21.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:13:21.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:13:21.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:13:21.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:13:21.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:13:21.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:13:21.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:13:21.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:13:21.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:13:21.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:13:21.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:21.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:21.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:13:21.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:13:21.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:13:21.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:13:21.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:13:21.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:21.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:21.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-03-09T16:13:21.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:13:21.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:21.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:13:21.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:13:21.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:13:21.950 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:13:21.950 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:21.950 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:21.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:13:21.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:13:21.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:13:22.031 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:22.032 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:22.032 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:13:22.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:13:22.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:13:22.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-09T16:13:22.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:13:22.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:13:22.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:13:22.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:13:22.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:13:22.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:13:22.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:13:22.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:13:22.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:13:22.953 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-09T16:13:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:13:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-09T16:13:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:13:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:13:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:13:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:13:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:13:22.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:13:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:13:22.965 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:13:22.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43181ef5-fdf2-4c82-8c63-f48bb19b345b 2024-03-09T16:13:22.966 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 43181ef5-fdf2-4c82-8c63-f48bb19b345b 2024-03-09T16:13:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 43181ef5-fdf2-4c82-8c63-f48bb19b345b' 2024-03-09T16:13:22.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:13:22.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiiuxlG9cdOhAAkbqHoiNAF8w4zz6ukr8FNA== 2024-03-09T16:13:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiiuxlG9cdOhAAkbqHoiNAF8w4zz6ukr8FNA=="}' 2024-03-09T16:13:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43181ef5-fdf2-4c82-8c63-f48bb19b345b -i td/osd-scrub-repair/0/new.json 2024-03-09T16:13:23.180 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:13:23.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:13:23.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiiuxlG9cdOhAAkbqHoiNAF8w4zz6ukr8FNA== --osd-uuid 43181ef5-fdf2-4c82-8c63-f48bb19b345b 2024-03-09T16:13:23.218 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.218+0000 7f00a3287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:23.231 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.230+0000 7f00a3287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:23.232 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.231+0000 7f00a3287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:23.232 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.232+0000 7f00a3287640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:13:23.232 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.232+0000 7f00a3287640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:13:23.232 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.232+0000 7f00a3287640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:13:23.323 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:23.322+0000 7f00a3287640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:13:28.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:13:28.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:13:28.824 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:13:28.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:13:28.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:13:29.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:13:29.177 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:13:29.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:13:29.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:13:29.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:13:29.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:13:29.203 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:29.202+0000 7f13e5468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:29.222 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:29.221+0000 7f13e5468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:29.225 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:29.222+0000 7f13e5468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:29.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:13:29.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:13:29.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:13:29.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:29.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:29.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:29.748 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:29.747+0000 7f13e5468640 -1 Falling back to public interface 2024-03-09T16:13:30.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:30.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:30.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:13:30.724 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:13:30.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:30.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:31.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:32.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:32.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:32.004 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:13:32.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:13:32.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:32.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:32.079 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:32.077+0000 7f13e5468640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:13:32.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:33.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:33.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:33.279 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:13:33.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:13:33.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:33.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:33.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:34.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:34.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:34.561 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:13:34.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:13:34.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:34.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:34.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:35.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:35.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:35.873 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:13:35.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:13:35.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:35.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:36.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:37.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:37.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:37.159 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:13:37.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:13:37.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:37.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:37.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:38.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:38.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:38.443 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:13:38.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:13:38.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:38.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:38.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:39.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:39.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:39.727 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:13:39.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:13:39.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:39.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:40.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:41.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:41.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:41.059 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:13:41.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:13:41.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:41.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:41.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:42.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:42.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:42.344 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:13:42.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:13:42.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:42.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:42.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:43.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:43.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:43.642 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:13:43.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:13:43.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:43.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:44.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:45.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:45.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:45.306 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:13:45.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:13:45.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:45.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:45.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:46.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:46.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:46.825 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:13:46.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:13:46.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:46.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:47.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:48.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:48.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:48.785 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:13:48.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:13:48.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:48.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:49.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:50.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:50.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:50.144 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:13:50.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:13:50.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:50.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:50.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:51.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:51.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:51.606 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:13:51.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:13:51.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:51.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:52.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:53.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:53.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:53.473 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:13:53.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:13:53.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:53.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:53.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:13:54.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:13:54.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:13:54.926 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:13:54.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:13:54.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:13:54.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:13:55.211 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/434156789,v1:127.0.0.1:6803/434156789] [v2:127.0.0.1:6804/434156789,v1:127.0.0.1:6805/434156789] exists,up 43181ef5-fdf2-4c82-8c63-f48bb19b345b 2024-03-09T16:13:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:13:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:13:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:13:55.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:13:55.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:13:55.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12223fe2-b248-49d3-86ed-f6347306942e 2024-03-09T16:13:55.218 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 12223fe2-b248-49d3-86ed-f6347306942e 2024-03-09T16:13:55.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 12223fe2-b248-49d3-86ed-f6347306942e' 2024-03-09T16:13:55.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:13:55.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDiuxlFuPSDRAA0uD59LsvX7GCM45PFPyFEw== 2024-03-09T16:13:55.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDiuxlFuPSDRAA0uD59LsvX7GCM45PFPyFEw=="}' 2024-03-09T16:13:55.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12223fe2-b248-49d3-86ed-f6347306942e -i td/osd-scrub-repair/1/new.json 2024-03-09T16:13:55.551 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:13:55.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:13:55.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDiuxlFuPSDRAA0uD59LsvX7GCM45PFPyFEw== --osd-uuid 12223fe2-b248-49d3-86ed-f6347306942e 2024-03-09T16:13:55.590 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.590+0000 7f97544a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:55.627 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.626+0000 7f97544a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:55.628 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.627+0000 7f97544a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:13:55.628 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.628+0000 7f97544a0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:13:55.628 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.628+0000 7f97544a0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:13:55.628 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.628+0000 7f97544a0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:13:55.703 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:13:55.702+0000 7f97544a0640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:14:01.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:14:01.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:14:01.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:14:01.681 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:14:01.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:14:02.040 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:14:02.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:14:02.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:14:02.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:14:02.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:14:02.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:14:02.066 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:02.066+0000 7f23c6171640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:02.107 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:02.107+0000 7f23c6171640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:02.108 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:02.108+0000 7f23c6171640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:02.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:14:02.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:14:02.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:14:02.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:14:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:14:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:02.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:14:02.318 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:14:02.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:02.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:02.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:02.624 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:02.624+0000 7f23c6171640 -1 Falling back to public interface 2024-03-09T16:14:03.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:03.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:03.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:14:03.592 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:14:03.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:03.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:03.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:04.874 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:14:04.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:14:04.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:04.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:04.948 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:04.947+0000 7f23c6171640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:14:05.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:06.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:06.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:06.166 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:14:06.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:14:06.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:06.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:06.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:07.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:07.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:07.466 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:14:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:14:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:07.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:08.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:08.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:08.776 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:14:08.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:14:08.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:08.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:09.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:10.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:10.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:10.117 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:14:10.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:14:10.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:10.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:10.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:11.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:11.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:11.486 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:14:11.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:14:11.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:11.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:11.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:12.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:12.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:12.768 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:14:12.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:14:12.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:12.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:13.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:14.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:14.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:14.129 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:14:14.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:14:14.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:14.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:14.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:15.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:15.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:15.475 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:14:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:14:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:15.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:16.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:16.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:16.756 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:14:16.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:14:16.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:16.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:17.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:18.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:18.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:18.303 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:14:18.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:14:18.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:18.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:18.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:19.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:19.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:19.978 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:14:19.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:14:19.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:19.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:20.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:21.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:21.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:21.551 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:14:21.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:14:21.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:21.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:21.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:22.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:22.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:22.893 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:14:22.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:14:22.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:22.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:23.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:24.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:24.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:24.279 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:14:24.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:14:24.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:24.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:25.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:25.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:25.724 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:14:25.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:14:25.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:25.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:26.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:27.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:27.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:27.098 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:14:27.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:14:27.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:27.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:14:27.412 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3396878501,v1:127.0.0.1:6811/3396878501] [v2:127.0.0.1:6812/3396878501,v1:127.0.0.1:6813/3396878501] exists,up 12223fe2-b248-49d3-86ed-f6347306942e 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:14:27.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:14:27.418 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:14:27.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5' 2024-03-09T16:14:27.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:14:27.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjiuxl2N/VGRAA/NeQlPrtPizsplHrqmF+og== 2024-03-09T16:14:27.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjiuxl2N/VGRAA/NeQlPrtPizsplHrqmF+og=="}' 2024-03-09T16:14:27.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:14:27.774 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:14:27.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:14:27.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjiuxl2N/VGRAA/NeQlPrtPizsplHrqmF+og== --osd-uuid 5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5 2024-03-09T16:14:27.813 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.813+0000 7feca200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:27.858 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.857+0000 7feca200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:27.859 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.858+0000 7feca200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:27.859 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.859+0000 7feca200f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:14:27.859 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.859+0000 7feca200f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:14:27.860 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.859+0000 7feca200f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:14:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:27.932+0000 7feca200f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:14:33.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:14:33.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:14:33.739 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:14:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:14:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:14:34.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:14:34.134 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:14:34.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:14:34.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:14:34.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:14:34.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:14:34.159 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:34.158+0000 7f9b68349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:34.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:34.160+0000 7f9b68349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:34.162 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:34.161+0000 7f9b68349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:14:34.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:14:34.406 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:14:34.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:34.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:34.678 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:34.677+0000 7f9b68349640 -1 Falling back to public interface 2024-03-09T16:14:34.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:35.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:35.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:35.680 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:14:35.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:14:35.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:35.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:35.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:36.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:36.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:36.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:14:36.959 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:14:36.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:36.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:37.013 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:14:37.012+0000 7f9b68349640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:14:37.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:38.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:38.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:38.275 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:14:38.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:14:38.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:38.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:38.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:39.557 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:14:39.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:14:39.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:39.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:39.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:40.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:40.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:40.840 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:14:40.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:14:40.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:40.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:41.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:42.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:42.129 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:14:42.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:42.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:14:42.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:42.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:42.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:43.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:43.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:43.483 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:14:43.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:14:43.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:43.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:43.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:44.875 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:14:44.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:14:44.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:44.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:45.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:46.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:46.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:46.157 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:14:46.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:14:46.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:46.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:46.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:47.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:47.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:47.481 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:14:47.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:14:47.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:47.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:47.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:48.824 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:14:48.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:14:48.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:48.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:49.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:50.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:50.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:50.410 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:14:50.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:14:50.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:50.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:51.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:52.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:52.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:52.090 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:14:52.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:14:52.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:52.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:52.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:53.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:53.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:53.935 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:14:53.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:14:53.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:53.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:54.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:55.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:55.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:55.527 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:14:55.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:14:55.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:55.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:56.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:57.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:57.094 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:14:57.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:57.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:14:57.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:57.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:57.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:14:58.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:14:58.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:14:58.479 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:14:58.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:14:58.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:14:58.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:14:59.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:00.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:00.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:00.208 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:15:00.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:15:00.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:00.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:15:00.480 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2218174807,v1:127.0.0.1:6819/2218174807] [v2:127.0.0.1:6820/2218174807,v1:127.0.0.1:6821/2218174807] exists,up 5b02b8b8-72e7-4cf4-8c34-93ff5577b8b5 2024-03-09T16:15:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:15:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:15:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:15:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:15:00.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:15:00.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:15:00.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:15:00.487 INFO:tasks.workunit.client.0.smithi146.stdout:add osd3 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:15:00.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1e55985e-7e14-4da0-95e2-64cd596b5e5b' 2024-03-09T16:15:00.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:15:00.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEi+xlUEHcHRAAxnDU/mh5fUvdf80iJNO33g== 2024-03-09T16:15:00.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEi+xlUEHcHRAAxnDU/mh5fUvdf80iJNO33g=="}' 2024-03-09T16:15:00.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e55985e-7e14-4da0-95e2-64cd596b5e5b -i td/osd-scrub-repair/3/new.json 2024-03-09T16:15:00.843 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:15:00.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T16:15:00.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEi+xlUEHcHRAAxnDU/mh5fUvdf80iJNO33g== --osd-uuid 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:15:00.883 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.882+0000 7f51a841b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:00.927 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.927+0000 7f51a841b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:00.928 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.928+0000 7f51a841b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:00.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.928+0000 7f51a841b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:15:00.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.928+0000 7f51a841b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:15:00.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:00.928+0000 7f51a841b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T16:15:01.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:01.001+0000 7f51a841b640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T16:15:07.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T16:15:07.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:15:07.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T16:15:07.035 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd3 key to auth repository 2024-03-09T16:15:07.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:15:07.387 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.3 2024-03-09T16:15:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T16:15:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:15:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:15:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:15:07.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:15:07.413 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:07.413+0000 7ff4b1ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:07.415 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:07.415+0000 7ff4b1ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:07.416 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:07.416+0000 7ff4b1ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:07.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:15:07.665 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:15:07.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:07.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:07.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:07.940 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:07.940+0000 7ff4b1ca0640 -1 Falling back to public interface 2024-03-09T16:15:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:08.935 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:15:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:15:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:09.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:10.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:10.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:10.218 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:15:10.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:15:10.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:10.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:10.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:10.365+0000 7ff4b1ca0640 -1 osd.3 0 log_to_monitors true 2024-03-09T16:15:10.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:11.576 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:15:11.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:15:11.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:11.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:11.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:12.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:12.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:12.860 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:15:12.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:15:12.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:12.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:13.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:14.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:14.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:14.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:15:14.199 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:15:14.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:14.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:14.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:15.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:15.513 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:15:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:15.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:15:15.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:15.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:15.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:16.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:16.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:16.795 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:15:16.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:15:16.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:16.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:17.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:18.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:18.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:18.214 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:15:18.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:15:18.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:18.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:18.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:19.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:19.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:19.503 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:15:19.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:15:19.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:19.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:19.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:20.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:20.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:20.785 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:15:20.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:15:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:21.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:22.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:22.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:22.172 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:15:22.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:15:22.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:22.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:22.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:23.525 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:15:23.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:15:23.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:23.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:24.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:25.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:25.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:25.470 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:15:25.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:15:25.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:25.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:26.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:27.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:27.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:27.331 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:15:27.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:15:27.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:27.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:28.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:29.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:29.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:29.067 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:15:29.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:15:29.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:29.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:29.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:30.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:30.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:30.969 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:15:30.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:15:30.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:30.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:31.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:32.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:32.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:32.400 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:15:32.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:15:32.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:32.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:32.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:33.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:33.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:33.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:15:33.892 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:15:33.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:33.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:15:34.169 INFO:tasks.workunit.client.0.smithi146.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3780248867,v1:127.0.0.1:6827/3780248867] [v2:127.0.0.1:6828/3780248867,v1:127.0.0.1:6829/3780248867] exists,up 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:15:34.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:15:34.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:15:34.174 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:15:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:15:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:15:34.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:15:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:15:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fc0f128-201a-46d5-bfca-02fcd75dd1d0 2024-03-09T16:15:34.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 5fc0f128-201a-46d5-bfca-02fcd75dd1d0' 2024-03-09T16:15:34.178 INFO:tasks.workunit.client.0.smithi146.stdout:add osd4 5fc0f128-201a-46d5-bfca-02fcd75dd1d0 2024-03-09T16:15:34.178 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:15:34.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmi+xlyv1jCxAApjeLAyeuWJnILC6E+iLi1Q== 2024-03-09T16:15:34.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmi+xlyv1jCxAApjeLAyeuWJnILC6E+iLi1Q=="}' 2024-03-09T16:15:34.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fc0f128-201a-46d5-bfca-02fcd75dd1d0 -i td/osd-scrub-repair/4/new.json 2024-03-09T16:15:34.618 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:15:34.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-09T16:15:34.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmi+xlyv1jCxAApjeLAyeuWJnILC6E+iLi1Q== --osd-uuid 5fc0f128-201a-46d5-bfca-02fcd75dd1d0 2024-03-09T16:15:34.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.658+0000 7f8ab7c25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:34.669 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.669+0000 7f8ab7c25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:34.670 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.670+0000 7f8ab7c25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:34.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.670+0000 7f8ab7c25640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:15:34.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.670+0000 7f8ab7c25640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:15:34.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.670+0000 7f8ab7c25640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T16:15:34.778 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:34.777+0000 7f8ab7c25640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-09T16:15:40.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-09T16:15:40.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:15:40.840 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd4 key to auth repository 2024-03-09T16:15:40.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T16:15:40.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:15:41.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T16:15:41.193 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.4 2024-03-09T16:15:41.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:15:41.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:15:41.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:15:41.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:15:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:41.218+0000 7fe8c9316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:41.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:41.220+0000 7fe8c9316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:41.222 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:41.221+0000 7fe8c9316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:41.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:15:41.492 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:15:41.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:41.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:41.756 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:41.756+0000 7fe8c9316640 -1 Falling back to public interface 2024-03-09T16:15:41.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:42.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:42.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:42.769 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:15:42.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:15:42.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:42.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:43.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:44.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:44.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:44.050 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:15:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:15:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:44.143 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:15:44.142+0000 7fe8c9316640 -1 osd.4 0 log_to_monitors true 2024-03-09T16:15:44.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:45.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:45.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:45.378 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:15:45.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:15:45.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:45.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:45.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:46.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:46.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:46.660 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:15:46.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:15:46.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:46.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:46.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:47.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:47.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:47.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:15:47.947 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:15:47.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:47.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:48.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:49.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:49.397 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:15:49.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:49.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:15:49.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:49.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:49.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:50.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:50.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:50.682 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:15:50.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:15:50.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:50.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:50.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:51.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:15:51.966 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:15:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:51.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:52.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:53.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:53.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:53.254 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:15:53.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:15:53.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:53.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:53.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:54.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:54.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:54.538 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:15:54.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:15:54.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:54.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:54.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:55.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:55.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:55.823 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:15:55.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:15:55.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:55.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:56.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:57.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:57.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:57.755 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:15:57.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:15:57.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:57.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:58.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:15:59.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:15:59.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:15:59.062 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:15:59.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:15:59.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:15:59.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:15:59.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:00.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:00.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:00.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:16:00.610 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:16:00.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:00.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:16:01.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:02.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:02.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:02.118 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:16:02.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:16:02.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:02.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:16:02.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:03.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:03.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:03.869 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:16:03.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:16:03.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:03.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:16:04.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:05.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:05.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:05.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:16:05.605 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:16:05.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:05.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:16:06.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:07.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:07.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:07.308 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:16:07.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:16:07.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:07.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T16:16:07.645 INFO:tasks.workunit.client.0.smithi146.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/660617471,v1:127.0.0.1:6835/660617471] [v2:127.0.0.1:6836/660617471,v1:127.0.0.1:6837/660617471] exists,up 5fc0f128-201a-46d5-bfca-02fcd75dd1d0 2024-03-09T16:16:07.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:16:07.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:16:07.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:16:07.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:16:07.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:16:07.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 146d0b0d-736c-48ea-8590-7d17db88f9e5' 2024-03-09T16:16:07.652 INFO:tasks.workunit.client.0.smithi146.stdout:add osd5 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:16:07.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:16:07.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHi+xlpUqjJxAA66w6k+M58mI+EoOTvHa6sA== 2024-03-09T16:16:07.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHi+xlpUqjJxAA66w6k+M58mI+EoOTvHa6sA=="}' 2024-03-09T16:16:07.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 146d0b0d-736c-48ea-8590-7d17db88f9e5 -i td/osd-scrub-repair/5/new.json 2024-03-09T16:16:08.001 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:16:08.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-09T16:16:08.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHi+xlpUqjJxAA66w6k+M58mI+EoOTvHa6sA== --osd-uuid 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:16:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.041+0000 7fa174222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:08.086 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.085+0000 7fa174222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.086+0000 7fa174222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.087+0000 7fa174222640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:16:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.087+0000 7fa174222640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:16:08.087 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.087+0000 7fa174222640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T16:16:08.160 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:08.160+0000 7fa174222640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-09T16:16:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-09T16:16:14.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:16:14.385 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd5 key to auth repository 2024-03-09T16:16:14.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T16:16:14.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:16:14.736 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.5 2024-03-09T16:16:14.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T16:16:14.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:16:14.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:16:14.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:16:14.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:16:14.764 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:14.763+0000 7ff42c4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:14.766 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:14.766+0000 7ff42c4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:14.767 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:14.767+0000 7ff42c4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:15.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T16:16:15.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:16:15.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:16:15.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:16:15.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:16:15.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:15.012 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:16:15.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:16:15.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:15.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:15.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:15.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:15.287+0000 7ff42c4f9640 -1 Falling back to public interface 2024-03-09T16:16:16.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:16.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:16.283 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:16:16.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:16:16.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:16.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:16.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:17.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:17.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:17.566 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:16:17.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:16:17.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:17.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:17.611 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:17.610+0000 7ff42c4f9640 -1 osd.5 0 log_to_monitors true 2024-03-09T16:16:17.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:18.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:18.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:18.849 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:16:18.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:16:18.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:18.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:19.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:20.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:20.134 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:16:20.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:16:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:20.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:21.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:21.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:21.468 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:16:21.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:16:21.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:21.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:21.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:22.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:22.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:22.756 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:16:22.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:16:22.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:22.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:23.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:24.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:24.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:24.039 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:16:24.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:16:24.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:24.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:24.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:25.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:25.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:25.343 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:16:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:16:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:25.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:25.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:26.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:26.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:26.639 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:16:26.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:16:26.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:26.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:26.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:27.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:27.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:27.921 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:16:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:16:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:28.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:29.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:29.205 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:16:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:16:29.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:29.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:29.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:30.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:30.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:30.500 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:16:30.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:16:30.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:30.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:30.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:31.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:31.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:31.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:16:31.877 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:16:31.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:31.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:32.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:33.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:33.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:33.402 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:16:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:16:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:33.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:34.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:34.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:34.975 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:16:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:16:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:34.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:35.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:36.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:36.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:36.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:16:36.786 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:16:36.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:36.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:37.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:38.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:38.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:38.335 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:16:38.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:16:38.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:38.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:39.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:40.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:40.143 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:16:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:16:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:40.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:41.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:41.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:41.582 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:16:41.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:16:41.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:41.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:16:41.872 INFO:tasks.workunit.client.0.smithi146.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1232224520,v1:127.0.0.1:6843/1232224520] [v2:127.0.0.1:6844/1232224520,v1:127.0.0.1:6845/1232224520] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:16:41.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:16:41.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:16:41.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=882bf7c7-f067-41a1-a04e-3b31c8fe2c7a 2024-03-09T16:16:41.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 882bf7c7-f067-41a1-a04e-3b31c8fe2c7a' 2024-03-09T16:16:41.880 INFO:tasks.workunit.client.0.smithi146.stdout:add osd6 882bf7c7-f067-41a1-a04e-3b31c8fe2c7a 2024-03-09T16:16:41.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:16:41.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpi+xl/585NRAAHEfzuirVM5mzj5M7WtMO2w== 2024-03-09T16:16:41.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpi+xl/585NRAAHEfzuirVM5mzj5M7WtMO2w=="}' 2024-03-09T16:16:41.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 882bf7c7-f067-41a1-a04e-3b31c8fe2c7a -i td/osd-scrub-repair/6/new.json 2024-03-09T16:16:42.228 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:16:42.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-09T16:16:42.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpi+xl/585NRAAHEfzuirVM5mzj5M7WtMO2w== --osd-uuid 882bf7c7-f067-41a1-a04e-3b31c8fe2c7a 2024-03-09T16:16:42.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.267+0000 7f0c12a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:42.321 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.320+0000 7f0c12a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:42.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.321+0000 7f0c12a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:42.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.322+0000 7f0c12a99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:16:42.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.322+0000 7f0c12a99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:16:42.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.322+0000 7f0c12a99640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T16:16:42.421 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:42.420+0000 7f0c12a99640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-09T16:16:48.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-09T16:16:48.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:16:48.395 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd6 key to auth repository 2024-03-09T16:16:48.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T16:16:48.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:16:48.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T16:16:48.747 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.6 2024-03-09T16:16:48.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:16:48.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:16:48.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:16:48.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:16:48.773 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:48.772+0000 7f0bff18c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:48.775 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:48.774+0000 7f0bff18c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:48.776 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:48.775+0000 7f0bff18c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:49.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:16:49.021 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:16:49.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:49.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:49.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:49.301 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:49.301+0000 7f0bff18c640 -1 Falling back to public interface 2024-03-09T16:16:50.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:50.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:50.298 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:16:50.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:16:50.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:50.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:51.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:51.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:51.579 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:16:51.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:16:51.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:51.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:51.671 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:16:51.670+0000 7f0bff18c640 -1 osd.6 0 log_to_monitors true 2024-03-09T16:16:51.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:52.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:52.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:52.880 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:16:52.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:16:52.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:52.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:54.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:54.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:54.186 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:16:54.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:16:54.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:54.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:54.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:55.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:55.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:55.678 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:16:55.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:16:55.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:55.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:55.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:56.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:56.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:56.962 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:16:56.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:16:56.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:56.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:57.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:58.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:58.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:58.297 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:16:58.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:16:58.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:58.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:16:58.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:16:59.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:16:59.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:16:59.763 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:16:59.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:16:59.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:16:59.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:00.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:01.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:01.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:01.044 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:17:01.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:17:01.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:01.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:01.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:02.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:02.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:02.339 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:17:02.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:17:02.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:02.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:02.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:03.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:03.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:03.632 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:17:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:17:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:03.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:04.914 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:17:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:17:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:05.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:06.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:06.444 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:17:06.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:06.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:17:06.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:06.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:07.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:08.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:08.322 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:17:08.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:17:08.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:08.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:08.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:09.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:09.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:09.649 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:17:09.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:17:09.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:09.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:10.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:11.452 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:17:11.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:17:11.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:11.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:11.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:12.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:12.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:12.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:17:12.928 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:17:12.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:12.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:13.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:14.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:14.531 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:17:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:17:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:14.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:15.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:15.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:15.932 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:17:15.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:17:15.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:15.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T16:17:16.215 INFO:tasks.workunit.client.0.smithi146.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1760620139,v1:127.0.0.1:6851/1760620139] [v2:127.0.0.1:6852/1760620139,v1:127.0.0.1:6853/1760620139] exists,up 882bf7c7-f067-41a1-a04e-3b31c8fe2c7a 2024-03-09T16:17:16.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:17:16.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:17:16.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T16:17:16.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-09T16:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=923491b5-38ff-450d-a950-26b81ad05f93 2024-03-09T16:17:16.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 923491b5-38ff-450d-a950-26b81ad05f93' 2024-03-09T16:17:16.225 INFO:tasks.workunit.client.0.smithi146.stdout:add osd7 923491b5-38ff-450d-a950-26b81ad05f93 2024-03-09T16:17:16.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:17:16.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMi+xlzHYMDhAAJWm/t0wQE61JRAcdRrxTrw== 2024-03-09T16:17:16.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMi+xlzHYMDhAAJWm/t0wQE61JRAcdRrxTrw=="}' 2024-03-09T16:17:16.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 923491b5-38ff-450d-a950-26b81ad05f93 -i td/osd-scrub-repair/7/new.json 2024-03-09T16:17:16.628 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:17:16.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-09T16:17:16.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMi+xlzHYMDhAAJWm/t0wQE61JRAcdRrxTrw== --osd-uuid 923491b5-38ff-450d-a950-26b81ad05f93 2024-03-09T16:17:16.668 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.667+0000 7f16eb60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:16.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.704+0000 7f16eb60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:16.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.705+0000 7f16eb60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:16.706 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.705+0000 7f16eb60f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:17:16.706 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.705+0000 7f16eb60f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:17:16.706 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.705+0000 7f16eb60f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T16:17:16.779 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:16.779+0000 7f16eb60f640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-09T16:17:22.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-09T16:17:22.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:17:22.798 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd7 key to auth repository 2024-03-09T16:17:22.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T16:17:22.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:17:23.154 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.7 2024-03-09T16:17:23.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T16:17:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:17:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:17:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:17:23.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:17:23.178 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:23.177+0000 7fb1941dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:23.180 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:23.179+0000 7fb1941dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:23.181 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:23.180+0000 7fb1941dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:17:23.444 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:17:23.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:23.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:23.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:23.699+0000 7fb1941dd640 -1 Falling back to public interface 2024-03-09T16:17:23.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:24.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:24.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:24.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:17:24.718 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:17:24.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:24.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:24.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:25.932 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:25.931+0000 7fb1941dd640 -1 osd.7 0 log_to_monitors true 2024-03-09T16:17:25.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:25.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:17:25.998 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:17:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:26.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:27.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:27.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:27.434 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:17:27.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:17:27.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:27.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:27.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:28.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:28.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:28.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:17:28.746 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:17:28.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:28.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:29.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:30.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:30.032 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:17:30.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:30.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:17:30.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:30.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:30.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:31.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:31.330 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:17:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:17:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:31.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:32.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:32.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:32.615 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:17:32.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:17:32.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:32.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:32.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:33.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:33.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:33.904 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:17:33.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:17:33.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:33.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:34.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:35.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:35.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:35.186 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:17:35.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:17:35.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:35.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:35.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:36.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:36.471 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:17:36.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:36.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:17:36.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:36.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:36.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:37.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:37.758 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:17:37.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:37.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:17:37.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:37.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:38.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:39.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:39.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:39.279 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:17:39.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:17:39.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:39.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:39.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:40.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:40.682 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:17:40.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:40.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:17:40.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:40.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:41.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:42.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:42.178 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:17:42.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:42.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:17:42.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:42.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:42.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:43.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:43.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:43.657 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:17:43.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:17:43.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:43.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:44.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:45.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:45.050 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:17:45.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:45.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:17:45.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:45.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:45.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:46.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:46.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:46.642 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:17:46.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:17:46.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:46.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:47.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:48.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:48.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:48.178 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:17:48.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:17:48.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:48.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T16:17:48.614 INFO:tasks.workunit.client.0.smithi146.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3435740684,v1:127.0.0.1:6859/3435740684] [v2:127.0.0.1:6860/3435740684,v1:127.0.0.1:6861/3435740684] exists,up 923491b5-38ff-450d-a950-26b81ad05f93 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:17:48.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:17:48.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:17:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:17:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e 2024-03-09T16:17:48.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e' 2024-03-09T16:17:48.622 INFO:tasks.workunit.client.0.smithi146.stdout:add osd8 fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e 2024-03-09T16:17:48.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:17:48.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsi+xlxg/hJRAAmZW5AWnWbm8+ZkgoBX0JCg== 2024-03-09T16:17:48.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsi+xlxg/hJRAAmZW5AWnWbm8+ZkgoBX0JCg=="}' 2024-03-09T16:17:48.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e -i td/osd-scrub-repair/8/new.json 2024-03-09T16:17:49.089 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:17:49.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-09T16:17:49.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsi+xlxg/hJRAAmZW5AWnWbm8+ZkgoBX0JCg== --osd-uuid fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e 2024-03-09T16:17:49.130 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.130+0000 7fbdac00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:49.140 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.140+0000 7fbdac00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:49.141 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.141+0000 7fbdac00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:49.142 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.141+0000 7fbdac00d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:17:49.142 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.141+0000 7fbdac00d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:17:49.142 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.141+0000 7fbdac00d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T16:17:49.240 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:49.240+0000 7fbdac00d640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-09T16:17:55.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-09T16:17:55.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:17:55.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-09T16:17:55.258 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd8 key to auth repository 2024-03-09T16:17:55.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:17:55.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-09T16:17:55.612 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.8 2024-03-09T16:17:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:17:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:17:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:17:55.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:17:55.638 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:55.637+0000 7fa0c6a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:55.639 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:55.639+0000 7fa0c6a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:55.641 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:55.640+0000 7fa0c6a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:17:55.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-09T16:17:55.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:17:55.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:17:55.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:55.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:17:56.155 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:56.155+0000 7fa0c6a1c640 -1 Falling back to public interface 2024-03-09T16:17:56.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:57.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:57.183 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:17:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:17:57.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:57.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:17:57.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:58.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:58.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:58.467 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:17:58.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:17:58.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:58.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:17:58.595 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:17:58.594+0000 7fa0c6a1c640 -1 osd.8 0 log_to_monitors true 2024-03-09T16:17:58.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:17:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:17:59.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:17:59.796 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:17:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:17:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:17:59.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:00.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:01.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:01.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:01.203 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:18:01.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:18:01.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:01.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:01.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:02.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:02.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:02.527 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:18:02.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:18:02.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:02.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:02.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:03.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:03.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:03.810 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:18:03.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:18:03.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:03.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:04.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:05.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:05.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:05.147 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:18:05.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:18:05.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:05.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:05.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:06.471 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:18:06.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:18:06.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:06.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:06.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:07.810 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:18:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:18:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:08.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:09.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:09.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:09.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:18:09.148 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:18:09.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:09.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:09.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:10.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:10.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:10.465 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:18:10.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:18:10.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:10.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:10.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:11.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:11.744 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:18:11.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:11.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:18:11.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:11.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:12.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:18:13.322 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:18:13.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:13.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:13.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:14.608 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:18:14.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:18:14.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:14.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:15.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:16.212 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:18:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:18:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:16.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:17.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:17.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:17.666 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:18:17.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:18:17.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:17.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:18.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:19.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:19.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:19.050 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:18:19.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:18:19.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:19.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:19.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:20.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:20.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:20.630 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:18:20.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:18:20.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:20.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:20.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:21.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:21.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:21.972 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:18:21.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:18:21.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:21.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T16:18:22.253 INFO:tasks.workunit.client.0.smithi146.stdout:osd.8 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/155157354,v1:127.0.0.1:6867/155157354] [v2:127.0.0.1:6868/155157354,v1:127.0.0.1:6869/155157354] exists,up fcbd95d9-813a-4edb-8a92-bfb3e6ccb83e 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:18:22.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:18:22.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:18:22.257 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:18:22.257 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:18:22.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6' 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stdout:add osd9 b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:18:22.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:18:22.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOi+xlUydLEBAA5wcQPQhGqJt2hGkC4DW3PA== 2024-03-09T16:18:22.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOi+xlUydLEBAA5wcQPQhGqJt2hGkC4DW3PA=="}' 2024-03-09T16:18:22.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 -i td/osd-scrub-repair/9/new.json 2024-03-09T16:18:22.718 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:18:22.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-09T16:18:22.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOi+xlUydLEBAA5wcQPQhGqJt2hGkC4DW3PA== --osd-uuid b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:18:22.759 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.758+0000 7f548ce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:22.802 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.802+0000 7f548ce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:22.803 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.803+0000 7f548ce11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.803+0000 7f548ce11640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:18:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.803+0000 7f548ce11640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:18:22.804 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.803+0000 7f548ce11640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T16:18:22.936 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:22.935+0000 7f548ce11640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-09T16:18:29.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-09T16:18:29.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:18:29.182 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd9 key to auth repository 2024-03-09T16:18:29.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-09T16:18:29.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:18:29.537 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.9 2024-03-09T16:18:29.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-09T16:18:29.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:18:29.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:18:29.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:18:29.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:18:29.564 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:29.563+0000 7f13ef013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:29.565+0000 7f13ef013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:29.567 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:29.566+0000 7f13ef013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:18:29.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-09T16:18:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:18:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:18:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:18:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:18:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:18:29.815 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:18:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:30.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:30.103 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:30.102+0000 7f13ef013640 -1 Falling back to public interface 2024-03-09T16:18:31.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:31.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:31.095 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:18:31.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:18:31.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:31.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:31.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:32.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:18:32.372 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:18:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:32.543 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:18:32.542+0000 7f13ef013640 -1 osd.9 0 log_to_monitors true 2024-03-09T16:18:32.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:33.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:33.655 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:18:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:18:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:33.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:33.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:34.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:34.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:34.940 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:18:34.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:18:34.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:34.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:35.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:36.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:36.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:36.249 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:18:36.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:18:36.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:36.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:36.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:37.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:37.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:37.602 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:18:37.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:18:37.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:37.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:37.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:38.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:38.888 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:18:38.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:18:38.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:38.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:39.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:40.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:18:40.175 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:18:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:40.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:40.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:41.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:41.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:41.485 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:18:41.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:18:41.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:41.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:41.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:42.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:42.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:42.767 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:18:42.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:18:42.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:42.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:43.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:44.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:45.338 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:18:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:18:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:46.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:47.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:47.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:47.130 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:18:47.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:18:47.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:47.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:47.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:48.828 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:18:48.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:18:48.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:48.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:49.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:50.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:50.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:50.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:18:50.485 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:18:50.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:50.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:51.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:52.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:52.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:52.126 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:18:52.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:18:52.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:52.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:52.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:53.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:53.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:53.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:18:53.752 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:18:53.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:53.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:54.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:55.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:55.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:55.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:18:55.211 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:18:55.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:55.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:55.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:18:56.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:18:56.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:18:56.591 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:18:56.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:18:56.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:18:56.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:18:56.903 INFO:tasks.workunit.client.0.smithi146.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2428384717,v1:127.0.0.1:6875/2428384717] [v2:127.0.0.1:6876/2428384717,v1:127.0.0.1:6877/2428384717] exists,up b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:18:56.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:18:56.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:18:56.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:18:56.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-09T16:18:56.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:18:57.170 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:18:57.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:18:57.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:18:57.584 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:18:57.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:18:58.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:19:02.087 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:19:02.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:19:02.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:19:02.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:19:02.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:02.497 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:02.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:19:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-03-09T16:19:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-03-09T16:19:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-03-09T16:19:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:02.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:19:02.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-03-09T16:19:02.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-03-09T16:19:02.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020' 2024-03-09T16:19:02.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:02.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:19:02.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542200 2024-03-09T16:19:02.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542200 2024-03-09T16:19:02.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200' 2024-03-09T16:19:02.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:02.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:19:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378671 2024-03-09T16:19:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378671 2024-03-09T16:19:03.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671' 2024-03-09T16:19:03.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:19:03.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215141 2024-03-09T16:19:03.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215141 2024-03-09T16:19:03.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141' 2024-03-09T16:19:03.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:19:03.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051620 2024-03-09T16:19:03.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051620 2024-03-09T16:19:03.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141 5-124554051620' 2024-03-09T16:19:03.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:19:03.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888093 2024-03-09T16:19:03.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888093 2024-03-09T16:19:03.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141 5-124554051620 6-146028888093' 2024-03-09T16:19:03.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:19:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724566 2024-03-09T16:19:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724566 2024-03-09T16:19:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141 5-124554051620 6-146028888093 7-167503724566' 2024-03-09T16:19:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:19:03.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561040 2024-03-09T16:19:03.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561040 2024-03-09T16:19:03.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141 5-124554051620 6-146028888093 7-167503724566 8-188978561040' 2024-03-09T16:19:03.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:03.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:19:03.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-03-09T16:19:03.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-03-09T16:19:03.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673020 2-60129542200 3-81604378671 4-103079215141 5-124554051620 6-146028888093 7-167503724566 8-188978561040 9-210453397509' 2024-03-09T16:19:03.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:03.681 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-03-09T16:19:03.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:03.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:19:03.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-03-09T16:19:03.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:03.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-03-09T16:19:03.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-03-09T16:19:03.684 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836551 2024-03-09T16:19:03.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:03.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-03-09T16:19:03.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:03.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-03-09T16:19:03.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:19:03.958 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-03-09T16:19:03.958 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:03.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-03-09T16:19:03.959 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949673020 2024-03-09T16:19:03.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-03-09T16:19:03.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:19:04.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-03-09T16:19:04.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:04.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542200 2024-03-09T16:19:04.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:04.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:19:04.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542200 2024-03-09T16:19:04.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:04.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542200 2024-03-09T16:19:04.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542200' 2024-03-09T16:19:04.268 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542200 2024-03-09T16:19:04.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:19:04.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542200 -lt 60129542200 2024-03-09T16:19:04.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:04.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378671 2024-03-09T16:19:04.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:04.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:19:04.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378671 2024-03-09T16:19:04.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:04.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378671 2024-03-09T16:19:04.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378671' 2024-03-09T16:19:04.541 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 81604378671 2024-03-09T16:19:04.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:19:04.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378670 -lt 81604378671 2024-03-09T16:19:04.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:19:05.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:19:05.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:19:06.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378672 -lt 81604378671 2024-03-09T16:19:06.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:06.107 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215141 2024-03-09T16:19:06.107 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:06.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:19:06.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215141 2024-03-09T16:19:06.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:06.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215141 2024-03-09T16:19:06.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215141' 2024-03-09T16:19:06.110 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.4 seq 103079215141 2024-03-09T16:19:06.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:19:06.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215141 -lt 103079215141 2024-03-09T16:19:06.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:06.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051620 2024-03-09T16:19:06.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:06.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:19:06.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051620 2024-03-09T16:19:06.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:06.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051620 2024-03-09T16:19:06.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051620' 2024-03-09T16:19:06.387 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.5 seq 124554051620 2024-03-09T16:19:06.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:19:06.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051620 -lt 124554051620 2024-03-09T16:19:06.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:06.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888093 2024-03-09T16:19:06.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:06.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:19:06.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888093 2024-03-09T16:19:06.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:06.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888093 2024-03-09T16:19:06.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888093' 2024-03-09T16:19:06.665 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.6 seq 146028888093 2024-03-09T16:19:06.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:19:06.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888093 -lt 146028888093 2024-03-09T16:19:06.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:06.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724566 2024-03-09T16:19:06.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:06.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:19:06.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724566 2024-03-09T16:19:06.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:06.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724566 2024-03-09T16:19:06.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724566' 2024-03-09T16:19:06.942 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.7 seq 167503724566 2024-03-09T16:19:06.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:19:07.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724567 -lt 167503724566 2024-03-09T16:19:07.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:07.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561040 2024-03-09T16:19:07.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:07.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:19:07.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561040 2024-03-09T16:19:07.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:07.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561040 2024-03-09T16:19:07.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561040' 2024-03-09T16:19:07.242 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.8 seq 188978561040 2024-03-09T16:19:07.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:19:07.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561040 -lt 188978561040 2024-03-09T16:19:07.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:07.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397509 2024-03-09T16:19:07.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:07.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:19:07.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397509 2024-03-09T16:19:07.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:07.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-03-09T16:19:07.515 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.9 seq 210453397509 2024-03-09T16:19:07.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397509' 2024-03-09T16:19:07.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:19:07.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397509 2024-03-09T16:19:07.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:19:07.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:07.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:19:08.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:19:08.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:19:08.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:19:08.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:08.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:08.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:19:08.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:19:08.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:19:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-03-09T16:19:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:19:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:19:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T16:19:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:19:08.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-09T16:19:09.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:19:09.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:19:09.891 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:19:09.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:19:10.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T16:19:10.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T16:19:11.468 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T16:19:11.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:19:11.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:19:11.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:19:11.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:19:11.480 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:19:11.481 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:19:11.481 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:19:11.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:19:11.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:19:11.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:19:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:19:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:19:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:19:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:19:11.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:19:11.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:19:11.959 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:11.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:19:12.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836554 2024-03-09T16:19:12.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836554 2024-03-09T16:19:12.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554' 2024-03-09T16:19:12.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:19:12.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-03-09T16:19:12.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-03-09T16:19:12.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022' 2024-03-09T16:19:12.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:19:12.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542203 2024-03-09T16:19:12.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542203 2024-03-09T16:19:12.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203' 2024-03-09T16:19:12.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:19:12.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378674 2024-03-09T16:19:12.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378674 2024-03-09T16:19:12.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674' 2024-03-09T16:19:12.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:19:12.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215144 2024-03-09T16:19:12.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215144 2024-03-09T16:19:12.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144' 2024-03-09T16:19:12.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:19:12.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051623 2024-03-09T16:19:12.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051623 2024-03-09T16:19:12.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144 5-124554051623' 2024-03-09T16:19:12.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:12.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:19:13.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888096 2024-03-09T16:19:13.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888096 2024-03-09T16:19:13.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144 5-124554051623 6-146028888096' 2024-03-09T16:19:13.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:13.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:19:13.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724569 2024-03-09T16:19:13.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724569 2024-03-09T16:19:13.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144 5-124554051623 6-146028888096 7-167503724569' 2024-03-09T16:19:13.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:13.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:19:13.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561043 2024-03-09T16:19:13.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561043 2024-03-09T16:19:13.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144 5-124554051623 6-146028888096 7-167503724569 8-188978561043' 2024-03-09T16:19:13.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:13.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:19:13.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397512 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397512 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673022 2-60129542203 3-81604378674 4-103079215144 5-124554051623 6-146028888096 7-167503724569 8-188978561043 9-210453397512' 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836554 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:19:13.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836554 2024-03-09T16:19:13.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836554 2024-03-09T16:19:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-03-09T16:19:13.386 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836554 2024-03-09T16:19:13.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:13.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836554 2024-03-09T16:19:13.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:13.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-03-09T16:19:13.659 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:13.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:19:13.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-03-09T16:19:13.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:13.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-03-09T16:19:13.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-03-09T16:19:13.662 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949673022 2024-03-09T16:19:13.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:19:13.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-03-09T16:19:13.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:13.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542203 2024-03-09T16:19:13.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:13.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:19:13.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542203 2024-03-09T16:19:13.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:13.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542203 2024-03-09T16:19:13.941 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542203 2024-03-09T16:19:13.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542203' 2024-03-09T16:19:13.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:19:14.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542203 -lt 60129542203 2024-03-09T16:19:14.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:14.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378674 2024-03-09T16:19:14.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:14.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:19:14.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378674 2024-03-09T16:19:14.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:14.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378674 2024-03-09T16:19:14.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378674' 2024-03-09T16:19:14.219 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.3 seq 81604378674 2024-03-09T16:19:14.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:19:14.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378674 -lt 81604378674 2024-03-09T16:19:14.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:14.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215144 2024-03-09T16:19:14.498 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:14.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:19:14.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215144 2024-03-09T16:19:14.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:14.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215144 2024-03-09T16:19:14.500 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.4 seq 103079215144 2024-03-09T16:19:14.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215144' 2024-03-09T16:19:14.501 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:19:14.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215144 -lt 103079215144 2024-03-09T16:19:14.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:14.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051623 2024-03-09T16:19:14.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:14.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:19:14.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051623 2024-03-09T16:19:14.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:14.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051623 2024-03-09T16:19:14.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051623' 2024-03-09T16:19:14.778 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.5 seq 124554051623 2024-03-09T16:19:14.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:19:15.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051623 -lt 124554051623 2024-03-09T16:19:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:15.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888096 2024-03-09T16:19:15.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:15.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:19:15.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888096 2024-03-09T16:19:15.057 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:15.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888096 2024-03-09T16:19:15.058 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.6 seq 146028888096 2024-03-09T16:19:15.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888096' 2024-03-09T16:19:15.058 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:19:15.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888096 -lt 146028888096 2024-03-09T16:19:15.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:15.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724569 2024-03-09T16:19:15.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:15.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:19:15.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724569 2024-03-09T16:19:15.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:15.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724569 2024-03-09T16:19:15.352 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.7 seq 167503724569 2024-03-09T16:19:15.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724569' 2024-03-09T16:19:15.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:19:15.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724570 -lt 167503724569 2024-03-09T16:19:15.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:15.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561043 2024-03-09T16:19:15.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:15.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:19:15.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561043 2024-03-09T16:19:15.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:15.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561043 2024-03-09T16:19:15.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561043' 2024-03-09T16:19:15.626 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.8 seq 188978561043 2024-03-09T16:19:15.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:19:15.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561043 -lt 188978561043 2024-03-09T16:19:15.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:15.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397512 2024-03-09T16:19:15.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:15.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:19:15.904 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397512 2024-03-09T16:19:15.904 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:15.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397512 2024-03-09T16:19:15.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397512' 2024-03-09T16:19:15.906 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.9 seq 210453397512 2024-03-09T16:19:15.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:19:16.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397512 -lt 210453397512 2024-03-09T16:19:16.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:19:16.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:16.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:19:16.534 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:19:16.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:19:16.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:19:16.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:16.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:17.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:19:17.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:19:17.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:19:17.559 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:19:17.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:19:18.080 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:19:18.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:19:18.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:19:18.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T16:19:18.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T16:19:18.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T16:19:18.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T16:19:18.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:19:18.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:19:18.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T16:19:18.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T16:19:18.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:19:18.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T16:19:18.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T16:19:18.967 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:19:18.967 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:9 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr:7' 2024-03-09T16:19:19.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:19:19.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:19:19.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:19:19.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:19:21.329 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:19:21.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:19:21.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:19:21.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:19:21.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:19:21.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:19:21.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:19:21.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:19:21.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:19:21.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:19:21.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:19:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:19:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:19:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:19:21.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:21.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:19:21.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:19:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:19:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:19:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:19:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:19:21.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:19:21.854 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.3 2024-03-09T16:19:21.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:19:21.854 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:19:21.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:19:21.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:19:21.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:19:21.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:19:21.877 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:21.875+0000 7fcfbd6f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:21.880 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:21.878+0000 7fcfbd6f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:21.881 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:21.879+0000 7fcfbd6f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:22.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:22.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:19:22.404 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:22.403+0000 7fcfbd6f9640 -1 Falling back to public interface 2024-03-09T16:19:22.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:23.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:19:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:23.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:19:23.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:24.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:19:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:24.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:19:24.762 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:24.760+0000 7fcfbd6f9640 -1 osd.3 68 log_to_monitors true 2024-03-09T16:19:24.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:19:25.998 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:25.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:25.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:19:26.348 INFO:tasks.workunit.client.0.smithi146.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [19,68) [v2:127.0.0.1:6826/2872041462,v1:127.0.0.1:6827/2872041462] [v2:127.0.0.1:6828/2872041462,v1:127.0.0.1:6829/2872041462] exists,up 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:19:26.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:19:26.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:19:26.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:19:26.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:19:26.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:26.711 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:19:26.712 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:19:26.712 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:19:26.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:19:26.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:26.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:19:26.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-03-09T16:19:26.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-03-09T16:19:26.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-03-09T16:19:26.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:26.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:19:26.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673026 2024-03-09T16:19:26.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673026 2024-03-09T16:19:26.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026' 2024-03-09T16:19:26.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:26.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:19:27.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542207 2024-03-09T16:19:27.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542207 2024-03-09T16:19:27.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207' 2024-03-09T16:19:27.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:19:27.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-09T16:19:27.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-09T16:19:27.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314' 2024-03-09T16:19:27.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:19:27.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215147 2024-03-09T16:19:27.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215147 2024-03-09T16:19:27.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147' 2024-03-09T16:19:27.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.324 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:19:27.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051627 2024-03-09T16:19:27.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051627 2024-03-09T16:19:27.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147 5-124554051627' 2024-03-09T16:19:27.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:19:27.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888100 2024-03-09T16:19:27.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888100 2024-03-09T16:19:27.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147 5-124554051627 6-146028888100' 2024-03-09T16:19:27.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:19:27.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724573 2024-03-09T16:19:27.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724573 2024-03-09T16:19:27.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147 5-124554051627 6-146028888100 7-167503724573' 2024-03-09T16:19:27.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:19:27.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561047 2024-03-09T16:19:27.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561047 2024-03-09T16:19:27.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147 5-124554051627 6-146028888100 7-167503724573 8-188978561047' 2024-03-09T16:19:27.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:27.740 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:19:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397516 2024-03-09T16:19:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397516 2024-03-09T16:19:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673026 2-60129542207 3-309237645314 4-103079215147 5-124554051627 6-146028888100 7-167503724573 8-188978561047 9-210453397516' 2024-03-09T16:19:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:27.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-03-09T16:19:27.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:19:27.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-03-09T16:19:27.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-03-09T16:19:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-03-09T16:19:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836558 2024-03-09T16:19:27.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:28.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-03-09T16:19:28.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:28.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673026 2024-03-09T16:19:28.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:28.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:19:28.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673026 2024-03-09T16:19:28.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:28.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673026 2024-03-09T16:19:28.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-03-09T16:19:28.123 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673026 2024-03-09T16:19:28.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:19:28.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673026 2024-03-09T16:19:28.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:28.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542207 2024-03-09T16:19:28.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:28.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:19:28.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542207 2024-03-09T16:19:28.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:28.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542207 2024-03-09T16:19:28.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542207' 2024-03-09T16:19:28.398 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542207 2024-03-09T16:19:28.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:19:28.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542206 -lt 60129542207 2024-03-09T16:19:28.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:19:29.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:19:29.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:19:29.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542207 -lt 60129542207 2024-03-09T16:19:29.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:29.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-03-09T16:19:29.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:29.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:19:29.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-03-09T16:19:29.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:29.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-09T16:19:29.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-03-09T16:19:29.957 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 309237645314 2024-03-09T16:19:29.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:19:30.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-03-09T16:19:30.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:30.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215147 2024-03-09T16:19:30.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:30.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:19:30.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215147 2024-03-09T16:19:30.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215147 2024-03-09T16:19:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215147' 2024-03-09T16:19:30.275 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 103079215147 2024-03-09T16:19:30.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:19:30.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215148 -lt 103079215147 2024-03-09T16:19:30.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:30.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051627 2024-03-09T16:19:30.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:30.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:19:30.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051627 2024-03-09T16:19:30.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:30.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051627 2024-03-09T16:19:30.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051627' 2024-03-09T16:19:30.552 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 124554051627 2024-03-09T16:19:30.552 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:19:30.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051627 -lt 124554051627 2024-03-09T16:19:30.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:30.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888100 2024-03-09T16:19:30.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:30.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:19:30.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888100 2024-03-09T16:19:30.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:30.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888100 2024-03-09T16:19:30.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888100' 2024-03-09T16:19:30.827 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 146028888100 2024-03-09T16:19:30.827 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:19:31.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888100 -lt 146028888100 2024-03-09T16:19:31.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:31.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724573 2024-03-09T16:19:31.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:31.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:19:31.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724573 2024-03-09T16:19:31.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724573 2024-03-09T16:19:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724573' 2024-03-09T16:19:31.125 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 167503724573 2024-03-09T16:19:31.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:19:31.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724574 -lt 167503724573 2024-03-09T16:19:31.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:31.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561047 2024-03-09T16:19:31.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:31.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:19:31.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561047 2024-03-09T16:19:31.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:31.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561047 2024-03-09T16:19:31.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561047' 2024-03-09T16:19:31.402 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 188978561047 2024-03-09T16:19:31.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:19:31.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561047 -lt 188978561047 2024-03-09T16:19:31.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:31.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397516 2024-03-09T16:19:31.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:31.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:19:31.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397516 2024-03-09T16:19:31.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:31.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397516 2024-03-09T16:19:31.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397516' 2024-03-09T16:19:31.676 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 210453397516 2024-03-09T16:19:31.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:19:31.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397516 -lt 210453397516 2024-03-09T16:19:31.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:19:31.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:31.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:19:32.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:19:32.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:19:32.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:19:32.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:32.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:32.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:19:32.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:19:33.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:19:33.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:19:33.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:19:33.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:19:09.756856+0000 2024-03-09T16:19:33.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:19:33.729 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:19:09.756856+0000 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:19:09.756856+0000 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:19:33.741 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:19:33.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:19:33.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:19:34.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:19:09.756856+0000 '>' 2024-03-09T16:19:09.756856+0000 2024-03-09T16:19:34.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:19:35.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:19:35.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:19:35.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:19:35.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:19:35.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:19:35.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:19:35.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:19:33.916433+0000 '>' 2024-03-09T16:19:09.756856+0000 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:19:35.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:19:35.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:19:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:19:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:19:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:19:35.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:19:35.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:19:35.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:19:35.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:19:35.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:19:37.638 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:19:37.638 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:19:37.638 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:19:37.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:37.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:19:37.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:19:37.912 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.9 2024-03-09T16:19:37.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:19:37.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:19:37.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:19:37.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:19:37.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:19:37.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:19:37.937 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:37.937+0000 7f7e8e3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:37.939+0000 7f7e8e3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:37.941 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:37.940+0000 7f7e8e3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:38.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:19:38.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:19:38.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:19:38.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:19:38.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:19:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:19:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:19:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:19:38.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:38.455+0000 7f7e8e3ac640 -1 Falling back to public interface 2024-03-09T16:19:38.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:19:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:39.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:19:39.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:40.759 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:40.758+0000 7f7e8e3ac640 -1 osd.9 73 log_to_monitors true 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:19:41.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 74 last_clean_interval [49,73) [v2:127.0.0.1:6874/3112235497,v1:127.0.0.1:6875/3112235497] [v2:127.0.0.1:6876/3112235497,v1:127.0.0.1:6877/3112235497] exists,up b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:19:42.586 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:19:42.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:19:42.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:19:42.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:19:42.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:19:42.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:19:42.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:19:42.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:19:42.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:19:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:19:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:43.273 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:19:43.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-03-09T16:19:43.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-03-09T16:19:43.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-03-09T16:19:43.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:19:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-03-09T16:19:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-03-09T16:19:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031' 2024-03-09T16:19:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.593 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:19:43.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542211 2024-03-09T16:19:43.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542211 2024-03-09T16:19:43.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211' 2024-03-09T16:19:43.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:19:43.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645318 2024-03-09T16:19:43.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645318 2024-03-09T16:19:43.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318' 2024-03-09T16:19:43.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:19:43.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215152 2024-03-09T16:19:43.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215152 2024-03-09T16:19:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152' 2024-03-09T16:19:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:43.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:19:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051631 2024-03-09T16:19:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051631 2024-03-09T16:19:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152 5-124554051631' 2024-03-09T16:19:44.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:44.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:19:44.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888104 2024-03-09T16:19:44.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888104 2024-03-09T16:19:44.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152 5-124554051631 6-146028888104' 2024-03-09T16:19:44.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:44.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:19:44.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724577 2024-03-09T16:19:44.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724577 2024-03-09T16:19:44.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152 5-124554051631 6-146028888104 7-167503724577' 2024-03-09T16:19:44.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:44.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:19:44.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561051 2024-03-09T16:19:44.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561051 2024-03-09T16:19:44.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152 5-124554051631 6-146028888104 7-167503724577 8-188978561051' 2024-03-09T16:19:44.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:44.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:19:44.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-09T16:19:44.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-09T16:19:44.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673031 2-60129542211 3-309237645318 4-103079215152 5-124554051631 6-146028888104 7-167503724577 8-188978561051 9-335007449090' 2024-03-09T16:19:44.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:44.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-03-09T16:19:44.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:44.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:19:44.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-03-09T16:19:44.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:44.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-03-09T16:19:44.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-03-09T16:19:44.469 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836562 2024-03-09T16:19:44.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:44.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836562 2024-03-09T16:19:44.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:19:45.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:19:45.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:46.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836562 2024-03-09T16:19:46.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:46.025 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-03-09T16:19:46.025 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:46.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:19:46.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-03-09T16:19:46.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:46.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-03-09T16:19:46.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-03-09T16:19:46.028 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673031 2024-03-09T16:19:46.028 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:19:46.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-03-09T16:19:46.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:46.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542211 2024-03-09T16:19:46.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:46.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:19:46.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542211 2024-03-09T16:19:46.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:46.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542211 2024-03-09T16:19:46.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542211' 2024-03-09T16:19:46.303 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542211 2024-03-09T16:19:46.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:19:46.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542211 -lt 60129542211 2024-03-09T16:19:46.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:46.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645318 2024-03-09T16:19:46.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:46.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:19:46.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645318 2024-03-09T16:19:46.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:46.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645318 2024-03-09T16:19:46.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645318' 2024-03-09T16:19:46.581 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 309237645318 2024-03-09T16:19:46.581 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:19:46.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645318 2024-03-09T16:19:46.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:46.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215152 2024-03-09T16:19:46.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:46.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:19:46.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215152 2024-03-09T16:19:46.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:46.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215152 2024-03-09T16:19:46.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215152' 2024-03-09T16:19:46.861 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 103079215152 2024-03-09T16:19:46.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:19:47.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215152 -lt 103079215152 2024-03-09T16:19:47.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:47.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051631 2024-03-09T16:19:47.137 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:47.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:19:47.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051631 2024-03-09T16:19:47.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:47.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051631 2024-03-09T16:19:47.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051631' 2024-03-09T16:19:47.140 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 124554051631 2024-03-09T16:19:47.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:19:47.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051632 -lt 124554051631 2024-03-09T16:19:47.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:47.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888104 2024-03-09T16:19:47.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:47.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:19:47.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888104 2024-03-09T16:19:47.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:47.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888104 2024-03-09T16:19:47.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888104' 2024-03-09T16:19:47.443 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 146028888104 2024-03-09T16:19:47.443 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:19:47.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888105 -lt 146028888104 2024-03-09T16:19:47.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:47.716 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724577 2024-03-09T16:19:47.716 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:47.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:19:47.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724577 2024-03-09T16:19:47.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:47.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724577 2024-03-09T16:19:47.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724577' 2024-03-09T16:19:47.719 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 167503724577 2024-03-09T16:19:47.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:19:47.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724578 -lt 167503724577 2024-03-09T16:19:47.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:47.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561051 2024-03-09T16:19:47.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:47.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:19:47.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561051 2024-03-09T16:19:47.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:47.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561051 2024-03-09T16:19:47.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561051' 2024-03-09T16:19:47.997 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 188978561051 2024-03-09T16:19:47.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:19:48.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561051 -lt 188978561051 2024-03-09T16:19:48.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:48.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449090 2024-03-09T16:19:48.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:48.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:19:48.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449090 2024-03-09T16:19:48.272 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:48.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-09T16:19:48.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-03-09T16:19:48.273 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 335007449090 2024-03-09T16:19:48.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:19:48.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449090 2024-03-09T16:19:48.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:19:48.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:48.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:48.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:19:48.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:19:48.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:19:49.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:19:49.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:19:49.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:19:49.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:19:49.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:19:49.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:19:49.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:19:49.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:19:49.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:19:49.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-09T16:19:49.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:19:49.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:19:49.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:19:49.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:19:49.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:19:49.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:19:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:19:51.719 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:19:52.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:19:52.241 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:19:52.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:19:52.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:19:52.243 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.5 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:19:52.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:19:52.272 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:52.269+0000 7f5adba23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:52.273 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:52.272+0000 7f5adba23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:52.274 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:52.273+0000 7f5adba23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:19:52.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:19:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:19:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:52.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:19:52.789 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:52.789+0000 7f5adba23640 -1 Falling back to public interface 2024-03-09T16:19:52.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:53.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:19:54.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:55.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:19:55.186 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:19:55.186+0000 7f5adba23640 -1 osd.5 80 log_to_monitors true 2024-03-09T16:19:55.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:19:56.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:19:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:19:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:19:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:19:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:19:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [29,80) [v2:127.0.0.1:6842/1847472094,v1:127.0.0.1:6843/1847472094] [v2:127.0.0.1:6844/1847472094,v1:127.0.0.1:6845/1847472094] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:19:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:19:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:19:56.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:19:56.722 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:19:56.723 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:19:56.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:19:56.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:19:56.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:19:56.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:19:57.172 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:19:57.173 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:19:57.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:19:57.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:19:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-03-09T16:19:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-03-09T16:19:57.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-03-09T16:19:57.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.520 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:19:57.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-03-09T16:19:57.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-03-09T16:19:57.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035' 2024-03-09T16:19:57.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:19:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542215 2024-03-09T16:19:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542215 2024-03-09T16:19:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215' 2024-03-09T16:19:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:19:57.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645322 2024-03-09T16:19:57.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645322 2024-03-09T16:19:57.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322' 2024-03-09T16:19:57.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:19:57.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215156 2024-03-09T16:19:57.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215156 2024-03-09T16:19:57.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156' 2024-03-09T16:19:57.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:57.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:19:58.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-09T16:19:58.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-09T16:19:58.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156 5-360777252866' 2024-03-09T16:19:58.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:58.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:19:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888108 2024-03-09T16:19:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888108 2024-03-09T16:19:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156 5-360777252866 6-146028888108' 2024-03-09T16:19:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:58.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:19:58.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724581 2024-03-09T16:19:58.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724581 2024-03-09T16:19:58.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156 5-360777252866 6-146028888108 7-167503724581' 2024-03-09T16:19:58.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:58.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:19:58.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561055 2024-03-09T16:19:58.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561055 2024-03-09T16:19:58.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156 5-360777252866 6-146028888108 7-167503724581 8-188978561055' 2024-03-09T16:19:58.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:19:58.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:19:58.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-03-09T16:19:58.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-03-09T16:19:58.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673035 2-60129542215 3-309237645322 4-103079215156 5-360777252866 6-146028888108 7-167503724581 8-188978561055 9-335007449094' 2024-03-09T16:19:58.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:19:58.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-03-09T16:19:58.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:19:58.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:19:58.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-03-09T16:19:58.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:19:58.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-03-09T16:19:58.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-03-09T16:19:58.527 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836566 2024-03-09T16:19:58.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:19:58.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836566 2024-03-09T16:19:58.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:19:59.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:19:59.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:00.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836566 2024-03-09T16:20:00.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:00.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-03-09T16:20:00.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:00.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:00.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-03-09T16:20:00.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:00.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-03-09T16:20:00.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-03-09T16:20:00.111 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673035 2024-03-09T16:20:00.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:00.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-03-09T16:20:00.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:00.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542215 2024-03-09T16:20:00.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:00.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:00.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542215 2024-03-09T16:20:00.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:00.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542215 2024-03-09T16:20:00.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542215' 2024-03-09T16:20:00.387 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542215 2024-03-09T16:20:00.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:00.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542215 -lt 60129542215 2024-03-09T16:20:00.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:00.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645322 2024-03-09T16:20:00.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:00.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:00.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645322 2024-03-09T16:20:00.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645322 2024-03-09T16:20:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645322' 2024-03-09T16:20:00.663 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 309237645322 2024-03-09T16:20:00.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:00.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645322 -lt 309237645322 2024-03-09T16:20:00.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:00.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215156 2024-03-09T16:20:00.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:00.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:00.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215156 2024-03-09T16:20:00.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:00.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215156 2024-03-09T16:20:00.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215156' 2024-03-09T16:20:00.942 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 103079215156 2024-03-09T16:20:00.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:01.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215156 -lt 103079215156 2024-03-09T16:20:01.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:01.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-03-09T16:20:01.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:01.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:01.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-03-09T16:20:01.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:01.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-09T16:20:01.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-03-09T16:20:01.219 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 360777252866 2024-03-09T16:20:01.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-03-09T16:20:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:01.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888108 2024-03-09T16:20:01.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:01.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:01.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888108 2024-03-09T16:20:01.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:01.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888108 2024-03-09T16:20:01.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888108' 2024-03-09T16:20:01.496 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 146028888108 2024-03-09T16:20:01.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:01.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888109 -lt 146028888108 2024-03-09T16:20:01.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:01.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724581 2024-03-09T16:20:01.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:01.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:01.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724581 2024-03-09T16:20:01.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:01.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724581 2024-03-09T16:20:01.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724581' 2024-03-09T16:20:01.770 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 167503724581 2024-03-09T16:20:01.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:02.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724582 -lt 167503724581 2024-03-09T16:20:02.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:02.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561055 2024-03-09T16:20:02.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:02.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:02.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561055 2024-03-09T16:20:02.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:02.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561055 2024-03-09T16:20:02.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561055' 2024-03-09T16:20:02.050 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 188978561055 2024-03-09T16:20:02.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:02.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561055 -lt 188978561055 2024-03-09T16:20:02.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:02.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449094 2024-03-09T16:20:02.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:02.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:02.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449094 2024-03-09T16:20:02.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:02.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-03-09T16:20:02.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-03-09T16:20:02.358 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 335007449094 2024-03-09T16:20:02.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:02.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449094 -lt 335007449094 2024-03-09T16:20:02.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:02.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:02.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:02.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:03.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:03.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:03.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:03.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:20:03.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:03.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:03.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:03.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:04.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:19:33.916433+0000 2024-03-09T16:20:04.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:20:04.378 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:20:04.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:19:33.916433+0000 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:19:33.916433+0000 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:04.390 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:04.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:19:33.916433+0000 '>' 2024-03-09T16:19:33.916433+0000 2024-03-09T16:20:04.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:05.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:05.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:05.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:05.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:05.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:05.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:05.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:05.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:19:33.916433+0000 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:20:05.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:05.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:05.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:05.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:05.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:06.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:06.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:20:08.143 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:20:08.143 INFO:tasks.workunit.client.0.smithi146.stdout:hinfo_key 2024-03-09T16:20:08.143 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:20:08.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:20:08.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:08.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:08.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:08.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:08.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:08.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:08.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:20:08.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:20:08.416 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.9 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:20:08.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:20:08.444 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:20:08.443+0000 7f937b16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:08.446 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:20:08.445+0000 7f937b16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:08.449 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:20:08.446+0000 7f937b16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:08.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:08.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:08.961 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:20:08.961+0000 7f937b16b640 -1 Falling back to public interface 2024-03-09T16:20:08.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:09.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:10.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:11.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:11.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:11.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:20:11.293 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:20:11.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:11.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:11.383 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:20:11.382+0000 7f937b16b640 -1 osd.9 86 log_to_monitors true 2024-03-09T16:20:11.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6874/910405136,v1:127.0.0.1:6875/910405136] [v2:127.0.0.1:6876/910405136,v1:127.0.0.1:6877/910405136] exists,up b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:20:12.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:20:12.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:20:12.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:20:12.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:4 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:5 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:6 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:7 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:8 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:9' 2024-03-09T16:20:13.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:20:13.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:13.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:20:13.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-03-09T16:20:13.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-03-09T16:20:13.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-03-09T16:20:13.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:13.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:20:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-03-09T16:20:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-03-09T16:20:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039' 2024-03-09T16:20:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:13.854 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:20:13.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542219 2024-03-09T16:20:13.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542219 2024-03-09T16:20:13.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219' 2024-03-09T16:20:13.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:13.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:20:14.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645326 2024-03-09T16:20:14.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645326 2024-03-09T16:20:14.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326' 2024-03-09T16:20:14.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.141 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:20:14.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215160 2024-03-09T16:20:14.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215160 2024-03-09T16:20:14.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160' 2024-03-09T16:20:14.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:20:14.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252870 2024-03-09T16:20:14.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252870 2024-03-09T16:20:14.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160 5-360777252870' 2024-03-09T16:20:14.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:20:14.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888112 2024-03-09T16:20:14.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888112 2024-03-09T16:20:14.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160 5-360777252870 6-146028888112' 2024-03-09T16:20:14.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:20:14.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724586 2024-03-09T16:20:14.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724586 2024-03-09T16:20:14.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160 5-360777252870 6-146028888112 7-167503724586' 2024-03-09T16:20:14.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:20:14.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561059 2024-03-09T16:20:14.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561059 2024-03-09T16:20:14.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160 5-360777252870 6-146028888112 7-167503724586 8-188978561059' 2024-03-09T16:20:14.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:14.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673039 2-60129542219 3-309237645326 4-103079215160 5-360777252870 6-146028888112 7-167503724586 8-188978561059 9-386547056642' 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-03-09T16:20:14.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:14.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:20:14.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-03-09T16:20:14.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:14.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-03-09T16:20:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-03-09T16:20:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836570 2024-03-09T16:20:14.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:15.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-03-09T16:20:15.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:20:16.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:20:16.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:16.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836570 2024-03-09T16:20:16.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:16.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-03-09T16:20:16.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:16.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:16.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-03-09T16:20:16.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:16.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-03-09T16:20:16.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-03-09T16:20:16.383 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949673039 2024-03-09T16:20:16.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:16.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673039 2024-03-09T16:20:16.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:16.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542219 2024-03-09T16:20:16.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:16.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:16.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542219 2024-03-09T16:20:16.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:16.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542219 2024-03-09T16:20:16.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542219' 2024-03-09T16:20:16.657 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 60129542219 2024-03-09T16:20:16.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:16.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542219 -lt 60129542219 2024-03-09T16:20:16.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:16.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645326 2024-03-09T16:20:16.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:16.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:16.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645326 2024-03-09T16:20:16.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:16.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645326 2024-03-09T16:20:16.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645326' 2024-03-09T16:20:16.934 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.3 seq 309237645326 2024-03-09T16:20:16.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:17.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645327 -lt 309237645326 2024-03-09T16:20:17.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:17.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215160 2024-03-09T16:20:17.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:17.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:17.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215160 2024-03-09T16:20:17.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:17.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215160 2024-03-09T16:20:17.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215160' 2024-03-09T16:20:17.214 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.4 seq 103079215160 2024-03-09T16:20:17.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215160 -lt 103079215160 2024-03-09T16:20:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:17.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252870 2024-03-09T16:20:17.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:17.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:17.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252870 2024-03-09T16:20:17.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252870 2024-03-09T16:20:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252870' 2024-03-09T16:20:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.5 seq 360777252870 2024-03-09T16:20:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:17.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252871 -lt 360777252870 2024-03-09T16:20:17.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:17.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888112 2024-03-09T16:20:17.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:17.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:17.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888112 2024-03-09T16:20:17.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:17.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888112 2024-03-09T16:20:17.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888112' 2024-03-09T16:20:17.770 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.6 seq 146028888112 2024-03-09T16:20:17.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:18.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888113 -lt 146028888112 2024-03-09T16:20:18.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:18.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724586 2024-03-09T16:20:18.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:18.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:18.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724586 2024-03-09T16:20:18.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:18.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724586 2024-03-09T16:20:18.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724586' 2024-03-09T16:20:18.046 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.7 seq 167503724586 2024-03-09T16:20:18.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724586 -lt 167503724586 2024-03-09T16:20:18.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:18.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561059 2024-03-09T16:20:18.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:18.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561059 2024-03-09T16:20:18.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:18.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561059 2024-03-09T16:20:18.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561059' 2024-03-09T16:20:18.363 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.8 seq 188978561059 2024-03-09T16:20:18.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:18.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561059 -lt 188978561059 2024-03-09T16:20:18.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:18.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-03-09T16:20:18.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:18.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:18.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-03-09T16:20:18.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:18.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-09T16:20:18.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-03-09T16:20:18.640 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.9 seq 386547056642 2024-03-09T16:20:18.640 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:18.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056642 2024-03-09T16:20:18.917 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:18.918 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:18.918 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:19.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:19.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:19.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:19.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:19.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:19.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:19.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:19.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:19.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:19.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:19.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:19.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:19.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:19.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:19.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:19.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:20:19.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268931"' 2024-03-09T16:20:19.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268931' 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268932"' 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268932' 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 268931 268932' 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268931 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268934: /' 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:20:19.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268936: /' 2024-03-09T16:20:22.955 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.956 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:22.956 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:20:22.956 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:22.957 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:20:22.957 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.957 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:22.958 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:20:22.958 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:22.958 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:20:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:22.959 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T16:20:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:22.960 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:20:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-09T16:20:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:268936: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T16:20:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:20:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:20:22.962 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:22.963 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:22.964 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:20:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:20:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:22.965 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:22.966 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:22.966 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:22.966 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:22.966 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:22.967 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:22.968 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:22.968 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:22.968 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:22.968 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:22.968 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: start osd.9 2024-03-09T16:20:22.969 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:20:22.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:20:22.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:22.977 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:22.978 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:22.980 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:22.981 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:22.982 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:20:22.982 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:20:22.982 INFO:tasks.workunit.client.0.smithi146.stderr:268934: start osd.5 2024-03-09T16:20:22.982 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:20:27.842 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2024-03-09T16:20:22.988+0000 7f8f66f47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2024-03-09T16:20:22.991+0000 7f8f66f47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2024-03-09T16:20:22.992+0000 7f8f66f47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:20:27.843 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 0 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2024-03-09T16:20:23.510+0000 7f8f66f47640 -1 Falling back to public interface 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 1 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2024-03-09T16:20:26.320+0000 7f8f66f47640 -1 osd.9 93 log_to_monitors true 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 3 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.845 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268936: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 93 last_c268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2024-03-09T16:20:23.006+0000 7fe1d9e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2024-03-09T16:20:23.009+0000 7fe1d9e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.846 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2024-03-09T16:20:23.010+0000 7fe1d9e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 0 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2024-03-09T16:20:23.536+0000 7fe1d9e19640 -1 Falling back to public interface 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.847 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 1 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2024-03-09T16:20:26.367+0000 7fe1d9e19640 -1 osd.5 92 log_to_monitors true 2024-03-09T16:20:27.848 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:27.849 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:27.849 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:20:27.849 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 3 2024-03-09T16:20:27.849 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:27.849 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:29.005 INFO:tasks.workunit.client.0.smithi146.stderr:268934: osd.5 up in weight 1 up_from 97 up_thru 84 down_at 93 last_lean_interval [90,93) [v2:127.0.0.1:6842/2140366113,v1:127.0.0.1:6843/2140366113] [v2:127.0.0.1:6844/2140366113,v1:127.0.0.1:6845/2140366113] exists,up b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:20:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:268936: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:20:29.007 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 1 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 2 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 3 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 4 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 5 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 6 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 7 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 8 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: 9' 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-03-09T16:20:29.008 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043' 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [84,92) [v2:127.0.0.1:6874/2692063602,v1:127.0.0.1:6875/2692063602] [v2:127.0.0.1:6876/2692063602,v1:127.0.0.1:6877/2692063602] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:20:29.009 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:20:29.010 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 1 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 2 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 3 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 4 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 5 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 6 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 7 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 8 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: 9' 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-03-09T16:20:29.011 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044' 2024-03-09T16:20:29.012 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542224 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542224 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224' 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645330 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645330 2024-03-09T16:20:30.260 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330' 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215164 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215164 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164' 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164 5-416611827714' 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:20:30.261 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888117 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888117 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164 5-416611827714 6-146028888117' 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724590 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724590 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164 5-416611827714 6-146028888117 7-167503724590' 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clon/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542223 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542223 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223' 2024-03-09T16:20:30.262 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645331 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645331 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331' 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215165 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215165 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165' 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-03-09T16:20:30.263 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165 5-416611827715' 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888116 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888116 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165 5-416611827715 6-146028888116' 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724591 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724591 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165 5-416611827715 6-146028888116 7-167503724591' 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.264 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:20:30.973 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clonee.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561064 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561064 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164 5-416611827714 6-146028888117 7-167503724590 8-188978561064' 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673044 2-60129542224 3-309237645330 4-103079215164 5-416611827714 6-146028888117 7-167503724590 8-188978561064 9-412316860419' 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:30.974 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.0 seq 21474836574 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836574 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-03-09T16:20:30.975 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.1 seq 42949673044 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673044 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542224 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:30.976 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:31.078 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone..client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561063 2024-03-09T16:20:31.078 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561063 2024-03-09T16:20:31.078 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165 5-416611827715 6-146028888116 7-167503724591 8-188978561063' 2024-03-09T16:20:31.078 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:31.078 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673043 2-60129542223 3-309237645331 4-103079215165 5-416611827715 6-146028888116 7-167503724591 8-188978561063 9-412316860418' 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.079 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.0 seq 21474836575 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836575 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.080 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.1 seq 42949673043 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673043 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542223 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.081 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.cclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542224 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542224 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542224' 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.2 seq 60129542224 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542225 -lt 60129542224 2024-03-09T16:20:31.901 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645330 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645330 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645330 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645330' 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.3 seq 309237645330 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645330 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215164 2024-03-09T16:20:31.902 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215164 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215164 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215164' 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.4 seq 103079215164 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215165 -lt 103079215164 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827714 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827714 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.903 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-03-09T16:20:31.983 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542223 2024-03-09T16:20:31.983 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.983 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542223 2024-03-09T16:20:31.983 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542223' 2024-03-09T16:20:31.983 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.2 seq 60129542223 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542225 -lt 60129542223 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645331 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645331 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645331 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.3 seq 309237645331 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:31.984 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645331 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215165 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215165 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215165 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215165' 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.4 seq 103079215165 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215165 -lt 103079215165 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827715 2024-03-09T16:20:31.985 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:31.986 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:31.986 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827715 2024-03-09T16:20:31.986 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:31.986 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-03-09T16:20:33.169 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiing osd.5 seq 416611827714' 2024-03-09T16:20:33.169 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.5 seq 416611827714 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888117 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888117 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888117 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888117' 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.6 seq 146028888117 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:33.170 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888118 -lt 146028888117 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724590 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724590 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724590 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724590' 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.7 seq 167503724590 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724591 -lt 167503724590 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561064 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.171 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561064 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561064 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561064' 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.8 seq 188978561064 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:33.172 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561065 -lt 188978561064 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 416611827715' 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.5 seq 416611827715 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888116 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888116 2024-03-09T16:20:33.201 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888116 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888116' 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.6 seq 146028888116 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888118 -lt 146028888116 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724591 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724591 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724591 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724591' 2024-03-09T16:20:33.202 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.7 seq 167503724591 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724591 -lt 167503724591 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561063 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561063 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561063 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561063' 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.8 seq 188978561063 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:33.203 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561065 -lt 188978561063 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: waiting osd.9 seq 412316860418 2024-03-09T16:20:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860418 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:268936: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:268936: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:34.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:g_stats: for s in $seqs 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860419 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860419 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: waiting osd.9 seq 412316860419 2024-03-09T16:20:34.552 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:34.553 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:268934: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:268934: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268932 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:20:34.554 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:20:34.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:20:34.555 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:34.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:34.830 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:35.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:35.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:20:35.401 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:20:35.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:35.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:35.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:35.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:36.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:36.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:36.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:36.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:36.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:36.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:36.688 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:36.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:36.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:37.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:37.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:37.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:37.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:37.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:37.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:37.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:38.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:38.239 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:39.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:39.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:39.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:39.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:39.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:40.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:40.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:40.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:40.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:40.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:40.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:40.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:40.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:04.906078+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:40.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:20:41.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:20:41.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:20:41.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:20:41.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:20:41.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:20:41.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:20:41.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:35.992511+0000 '>' 2024-03-09T16:20:04.906078+0000 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272362"' 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272362' 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:42.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272364"' 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272364' 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272362 272364' 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272362 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272365: /' 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:42.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272367: /' 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:44.799 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: _ 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: hinfo_key 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: snapset 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:44.801 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:44.802 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:44.803 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T16:20:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T16:20:44.804 INFO:tasks.workunit.client.0.smithi146.stderr:272367: start osd.9 2024-03-09T16:20:44.912 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:44.913 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: _ 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: hinfo_key 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: snapset 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:20:44.914 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:44.915 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:44.916 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:20:44.917 INFO:tasks.workunit.client.0.smithi146.stderr:272365: start osd.5 2024-03-09T16:20:49.371 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:20:49.371 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T16:20:49.371 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T16:20:49.371 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2024-03-09T16:20:44.830+0000 7f009d393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2024-03-09T16:20:44.832+0000 7f009d393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2024-03-09T16:20:44.833+0000 7f009d393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.372 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:20:49.373 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 0 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2024-03-09T16:20:45.351+0000 7f009d393640 -1 Falling back to public interface 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.374 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 1 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.375 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2024-03-09T16:20:47.973+0000 7f009d393640 -1 osd.9 100 log_to_monitors true 2024-03-09T16:20:49.376 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 3 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272367: osd.9 up in weight 1 up_from 105 up_thru 0 down_overy_ops 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:20:49.377 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2024-03-09T16:20:44.941+0000 7f6a7d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2024-03-09T16:20:44.943+0000 7f6a7d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2024-03-09T16:20:44.944+0000 7f6a7d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 0 2024-03-09T16:20:49.378 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2024-03-09T16:20:45.466+0000 7f6a7d011640 -1 Falling back to public interface 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 1 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:20:49.379 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2024-03-09T16:20:48.236+0000 7f6a7d011640 -1 osd.5 100 log_to_monitors true 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 3 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:20:49.380 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: osd.5 up in weight 1 up_from 105 up_thru 97 down_at 101 last_clean_interval [97,100) [v2:127.0.0.1:6874/590602321,v1:127.0.0.1:6875/590602321] [v2:127.0.0.1:6876/590602321,v1:127.0.0.1:6877/590602321] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:20:50.460 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:20:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 1 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 2 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 3 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 4 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 5 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 6 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 7 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 8 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: 9' 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-03-09T16:20:50.462 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049' 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.at 101 last_clean_interval [96,100) [v2:127.0.0.1:6842/2380670119,v1:127.0.0.1:6843/2380670119] [v2:127.0.0.1:6844/2380670119,v1:127.0.0.1:6845/2380670119] exists,up b4b1ba1e-58ae-43fc-8011-4c6327d9bbf6 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:20:50.463 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 1 2024-03-09T16:20:50.464 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 2 2024-03-09T16:20:50.465 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 3 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 4 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 5 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 6 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 7 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 8 2024-03-09T16:20:50.466 INFO:tasks.workunit.client.0.smithi146.stderr:272367: 9' 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050' 2024-03-09T16:20:50.467 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542230 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542230 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230' 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645338 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645338 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338' 2024-03-09T16:20:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215171 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215171 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171' 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171 5-450971566083' 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888124 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888124 2024-03-09T16:20:52.035 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171 5-450971566083 6-146028888124' 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724597 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724597 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171 5-450971566083 6-146028888124 7-167503724597' 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542229 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542229 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229' 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:20:52.036 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645337 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645337 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337' 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215170 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215170 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170' 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170 5-450971566082' 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888123 2024-03-09T16:20:52.037 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888123 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170 5-450971566082 6-146028888123' 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724596 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724596 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170 5-450971566082 6-146028888123 7-167503724596' 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.038 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cepcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561069 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561069 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171 5-450971566083 6-146028888124 7-167503724597 8-188978561069' 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673050 2-60129542230 3-309237645338 4-103079215171 5-450971566083 6-146028888124 7-167503724597 8-188978561069 9-450971566082' 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.766 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.0 seq 21474836582 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:52.767 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.1 seq 42949673050 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542230 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.768 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:52.771 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cehtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561070 2024-03-09T16:20:52.771 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561070 2024-03-09T16:20:52.771 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170 5-450971566082 6-146028888123 7-167503724596 8-188978561070' 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673049 2-60129542229 3-309237645337 4-103079215170 5-450971566082 6-146028888123 7-167503724596 8-188978561070 9-450971566083' 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.0 seq 21474836581 2024-03-09T16:20:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.1 seq 42949673049 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673049 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542229 2024-03-09T16:20:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:52.774 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:20:53.702 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542230 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542230 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542230' 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.2 seq 60129542230 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542231 -lt 60129542230 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645338 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645338 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.703 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645338 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645338' 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.3 seq 309237645338 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645338 -lt 309237645338 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215171 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215171 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215171 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215171' 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.4 seq 103079215171 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215172 -lt 103079215171 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566083 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566083 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-03-09T16:20:53.748 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_statsest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542229 2024-03-09T16:20:53.748 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.748 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542229 2024-03-09T16:20:53.748 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542229' 2024-03-09T16:20:53.748 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.2 seq 60129542229 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542231 -lt 60129542229 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645337 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645337 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645337 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645337' 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.3 seq 309237645337 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645338 -lt 309237645337 2024-03-09T16:20:53.749 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215170 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215170 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215170 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215170' 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.4 seq 103079215170 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215172 -lt 103079215170 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566082 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:53.750 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:20:53.751 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566082 2024-03-09T16:20:53.751 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:53.751 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-09T16:20:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: : echo 'waiting osd.5 seq 450971566083' 2024-03-09T16:20:54.920 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.5 seq 450971566083 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888124 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888124 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888124 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888124' 2024-03-09T16:20:54.921 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.6 seq 146028888124 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888124 -lt 146028888124 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724597 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724597 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724597 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724597' 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.7 seq 167503724597 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:54.922 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724597 -lt 167503724597 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561069 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561069 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561069 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561069' 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.8 seq 188978561069 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:54.923 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561071 -lt 188978561069 2024-03-09T16:20:54.994 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:echo 'waiting osd.5 seq 450971566082' 2024-03-09T16:20:54.994 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.5 seq 450971566082 2024-03-09T16:20:54.994 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:20:54.994 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566082 2024-03-09T16:20:54.994 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888123 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888123 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888123 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888123' 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.6 seq 146028888123 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888124 -lt 146028888123 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724596 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.995 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724596 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724596 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724596' 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.7 seq 167503724596 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724597 -lt 167503724596 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561070 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561070 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561070 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561070' 2024-03-09T16:20:54.996 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.8 seq 188978561070 2024-03-09T16:20:54.997 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:20:54.997 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561071 -lt 188978561070 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212210: flush_pg_stats: for s in $seqs 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566082 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566082 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: waiting osd.9 seq 450971566082 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566082 2024-03-09T16:20:56.252 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:56.253 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:56.254 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:56.254 INFO:tasks.workunit.client.0.smithi146.stderr:272367: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:56.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:0: flush_pg_stats: for s in $seqs 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566083 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566083 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: waiting osd.9 seq 450971566083 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566083 2024-03-09T16:20:56.333 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:20:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:272365: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:272365: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272364 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:20:56.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:20:56.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:20:56.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-09T16:20:56.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T16:20:56.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T16:20:56.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275595"' 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275595' 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275597"' 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275597' 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T16:20:56.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 275595 275597' 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275595 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275598: /' 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:56.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275600: /' 2024-03-09T16:20:59.221 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:59.222 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:59.223 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:20:59.224 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:20:59.225 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:59.226 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:59.226 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:59.226 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:59.226 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:59.227 INFO:tasks.workunit.client.0.smithi146.stderr:275600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:59.228 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:59.229 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:59.229 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:20:59.229 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:20:59.229 INFO:tasks.workunit.client.0.smithi146.stderr:275600: start osd.5 2024-03-09T16:20:59.229 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:20:59.341 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:20:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:20:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:20:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:20:59.345 INFO:tasks.workunit.client.0.smithi146.stderr:275598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:20:59.346 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:20:59.347 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:20:59.347 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:20:59.347 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:20:59.347 INFO:tasks.workunit.client.0.smithi146.stderr:275598: start osd.3 2024-03-09T16:20:59.347 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2024-03-09T16:20:59.252+0000 7fd38d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2024-03-09T16:20:59.255+0000 7fd38d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2024-03-09T16:20:59.256+0000 7fd38d011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:21:03.914 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 0 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2024-03-09T16:20:59.773+0000 7fd38d011640 -1 Falling back to public interface 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 1 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.915 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2024-03-09T16:21:02.390+0000 7fd38d011640 -1 osd.5 108 log_to_monitors true 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 3 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.916 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:03.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: osd.5 up in weight 1 up_from 112 up_thru 112 down_at 109 l275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2024-03-09T16:20:59.370+0000 7f530e019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2024-03-09T16:20:59.372+0000 7f530e019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2024-03-09T16:20:59.373+0000 7f530e019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:21:03.939 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 0 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2024-03-09T16:20:59.888+0000 7f530e019640 -1 Falling back to public interface 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 1 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.940 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2024-03-09T16:21:02.692+0000 7f530e019640 -1 osd.3 108 log_to_monitors true 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 3 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:03.941 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:04.935 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [105,108) [v2:127.0.0.1:6826/2957220584,v1:127.0.0.1:6827/2957220584] [v2:127.0.0.1:6828/2957220584,v1:127.0.0.1:6829/2957220584] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:21:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 1 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 2 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 3 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 4 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 5 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 6 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 7 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 8 2024-03-09T16:21:04.937 INFO:tasks.workunit.client.0.smithi146.stderr:275600: 9' 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054' 2024-03-09T16:21:04.938 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.131 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.cliers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:06.131 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:06.131 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:06.131 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:21:06.131 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 4 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: osd.3 up in weight 1 up_from 115 up_thru 115 down_at 109 last_clean_interval [72,108) [v2:127.0.0.1:6874/1920825522,v1:127.0.0.1:6875/1920825522] [v2:127.0.0.1:6876/1920825522,v1:127.0.0.1:6877/1920825522] exists,up 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:21:06.132 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 1 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 2 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 3 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 4 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 5 2024-03-09T16:21:06.133 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 6 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 7 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 8 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: 9' 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-03-09T16:21:06.134 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.363 INFO:tasks.workunit.client.0.smithi146.stderr:275nt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:21:06.363 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542234 2024-03-09T16:21:06.363 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542234 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234' 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042' 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215175 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215175 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175' 2024-03-09T16:21:06.364 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175 5-481036337154' 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888128 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888128 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175 5-481036337154 6-146028888128' 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724601 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724601 2024-03-09T16:21:06.365 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175 5-481036337154 6-146028888128 7-167503724601' 2024-03-09T16:21:06.366 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:06.366 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:21:07.018 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtes598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055' 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542236 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542236 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236' 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239043 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239043 2024-03-09T16:21:07.019 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043' 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215176 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215176 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176' 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337155 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337155 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176 5-481036337155' 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888129 2024-03-09T16:21:07.020 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888129 2024-03-09T16:21:07.021 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176 5-481036337155 6-146028888129' 2024-03-09T16:21:07.021 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.021 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: t/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561074 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561074 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175 5-481036337154 6-146028888128 7-167503724601 8-188978561074' 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:21:07.050 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566087 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566087 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673054 2-60129542234 3-493921239042 4-103079215175 5-481036337154 6-146028888128 7-167503724601 8-188978561074 9-450971566087' 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.0 seq 21474836586 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:07.051 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.1 seq 42949673054 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542234 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:07.052 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:21:07.562 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/flush_pg_stats: seq=167503724602 2024-03-09T16:21:07.562 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724602 2024-03-09T16:21:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176 5-481036337155 6-146028888129 7-167503724602' 2024-03-09T16:21:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:21:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561076 2024-03-09T16:21:07.563 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561076 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176 5-481036337155 6-146028888129 7-167503724602 8-188978561076' 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-03-09T16:21:07.564 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673055 2-60129542236 3-493921239043 4-103079215176 5-481036337155 6-146028888129 7-167503724602 8-188978561076 9-450971566088' 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:07.565 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.0 seq 21474836587 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:07.566 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-03-09T16:21:07.567 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-03-09T16:21:07.568 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.1 seq 42949673055 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flusclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542234 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542234 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542234' 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.2 seq 60129542234 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542236 -lt 60129542234 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.039 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239042 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239042 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.3 seq 493921239042 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239042 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215175 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215175 2024-03-09T16:21:08.040 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215175 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215175' 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.4 seq 103079215175 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215176 -lt 103079215175 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337154 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337154 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.041 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-03-09T16:21:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echoh_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:21:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673055 2024-03-09T16:21:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542236 2024-03-09T16:21:08.824 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542236 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542236 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542236' 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.2 seq 60129542236 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542236 -lt 60129542236 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239043 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239043 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.825 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239043 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.3 seq 493921239043 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239043 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215176 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215176 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215176 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215176' 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.4 seq 103079215176 2024-03-09T16:21:08.826 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:21:08.827 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215176 -lt 103079215176 2024-03-09T16:21:08.827 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:08.827 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-481036337155 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubu 'waiting osd.5 seq 481036337154' 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.5 seq 481036337154 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337154 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888128 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888128 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888128 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888128' 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.6 seq 146028888128 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888129 -lt 146028888128 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724601 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:21:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724601 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724601 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724601' 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.7 seq 167503724601 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724602 -lt 167503724601 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561074 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561074 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561074 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561074' 2024-03-09T16:21:09.376 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.8 seq 188978561074 2024-03-09T16:21:09.377 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:21:09.377 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561076 -lt 188978561074 2024-03-09T16:21:09.811 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.811 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:21:09.811 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-481036337155 2024-03-09T16:21:09.811 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.811 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337155 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 481036337155' 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.5 seq 481036337155 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337155 -lt 481036337155 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888129 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888129 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888129 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888129' 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.6 seq 146028888129 2024-03-09T16:21:09.812 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888129 -lt 146028888129 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724602 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724602 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724602 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724602' 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.7 seq 167503724602 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724602 -lt 167503724602 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561076 2024-03-09T16:21:09.813 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:09.814 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:21:09.814 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561076 2024-03-09T16:21:10.666 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stalush_pg_stats: for s in $seqs 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566087 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566087 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566087 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566087' 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: waiting osd.9 seq 450971566087 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566087 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:10.667 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:275600: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:21:10.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:21:11.507 INFO:tasks.workunit.client.0.smithi146.stderr:ts: cut -d - -f 2 2024-03-09T16:21:11.507 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561076 2024-03-09T16:21:11.507 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561076' 2024-03-09T16:21:11.507 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.8 seq 188978561076 2024-03-09T16:21:11.508 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:21:11.508 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561076 -lt 188978561076 2024-03-09T16:21:11.508 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:11.508 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566088 2024-03-09T16:21:11.508 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:11.509 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:21:11.509 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566088 2024-03-09T16:21:11.509 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:11.509 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-03-09T16:21:11.509 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-03-09T16:21:11.510 INFO:tasks.workunit.client.0.smithi146.stderr:275598: waiting osd.9 seq 450971566088 2024-03-09T16:21:11.510 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:21:11.510 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566088 2024-03-09T16:21:11.510 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:21:11.510 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:21:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:21:11.512 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:11.513 INFO:tasks.workunit.client.0.smithi146.stderr:275598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:11.513 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:21:11.513 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:21:11.513 INFO:tasks.workunit.client.0.smithi146.stderr:275598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:21:11.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:21:11.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:21:11.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275597 2024-03-09T16:21:11.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:21:11.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:21:11.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:21:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T16:21:11.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:21:11.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:21:11.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:21:12.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:12.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:21:12.352 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:21:12.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:21:12.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:21:12.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:21:12.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:21:12.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:21:12.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:21:12.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:35.992511+0000 '>' 2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:12.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:21:13.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:21:13.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:21:13.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:21:13.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:21:13.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:21:13.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:21:13.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:21:13.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:20:35.992511+0000 '>' 2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:13.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:21:14.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:21:14.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:21:14.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:21:14.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:21:14.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:21:14.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:21:14.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:21:15.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:21:12.740583+0000 '>' 2024-03-09T16:20:35.992511+0000 2024-03-09T16:21:15.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:21:15.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T16:21:15.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:21:15.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278975"' 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278975' 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278977"' 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278977' 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 278975 278977' 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:21:15.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278975 2024-03-09T16:21:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278978: /' 2024-03-09T16:21:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:21:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278980: /' 2024-03-09T16:21:17.954 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:21:17.955 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: _ 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: hinfo_key 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: snapset 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T16:21:17.956 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:21:17.957 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:21:17.957 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T16:21:17.957 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:21:17.957 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T16:21:17.957 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:21:17.958 INFO:tasks.workunit.client.0.smithi146.stderr:278978: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:21:17.959 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T16:21:17.960 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T16:21:17.960 INFO:tasks.workunit.client.0.smithi146.stderr:278978: start osd.3 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:21:18.228 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: _ 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: hinfo_key 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: snapset 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T16:21:18.229 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T16:21:18.230 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:21:18.231 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T16:21:18.232 INFO:tasks.workunit.client.0.smithi146.stderr:278980: start osd.5 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2024-03-09T16:21:17.986+0000 7efda4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2024-03-09T16:21:17.988+0000 7efda4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2024-03-09T16:21:17.989+0000 7efda4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T16:21:22.106 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 0 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2024-03-09T16:21:18.504+0000 7efda4ebd640 -1 Falling back to public interface 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 1 2024-03-09T16:21:22.107 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2024-03-09T16:21:20.908+0000 7efda4ebd640 -1 osd.3 116 log_to_monitors true 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 3 2024-03-09T16:21:22.108 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2024-03-09T16:21:18.256+0000 7f97902b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2024-03-09T16:21:18.277+0000 7f97902b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2024-03-09T16:21:18.278+0000 7f97902b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 0 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2024-03-09T16:21:18.803+0000 7f97902b5640 -1 Falling back to public interface 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 1 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2024-03-09T16:21:21.773+0000 7f97902b5640 -1 osd.5 116 log_to_monitors true 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:21:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 3 2024-03-09T16:21:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:22.954 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:23.402 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: osd.3 up in weight 1 up_from 120 up_thru 115 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6826/1321906300,v1:127.0.0.1:6827/1321906300] [v2:127.0.0.1:6828/1321906300,v1:127.0.0.1:6829/1321906300] exists,up 1e55985e-7e14-4da0-95e2-64cd596b5e5b 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:21:23.403 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 1 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 2 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 3 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 4 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 5 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 6 2024-03-09T16:21:23.404 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 7 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 8 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: 9' 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-03-09T16:21:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060' 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542240 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542240 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240' 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522' 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.698 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215181 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215181 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181' 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181 5-528280977410' 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888133 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888133 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181 5-528280977410 6-146028888133' 2024-03-09T16:21:24.699 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:24.700 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:21:24.700 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724607 2024-03-09T16:21:24.700 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724607 2024-03-09T16:21:24.700 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181 5-528280977410 6-146028888133 7-167503724607' 2024-03-09T16:21:24.700 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 4 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: osd.5 up in weight 1 up_from 123 up_thru 123 down_at 117 last_clean_interval [112,116) [v2:127.0.0.1:6874/3954747856,v1:127.0.0.1:6875/3954747856] [v2:127.0.0.1:6876/3954747856,v1:127.0.0.1:6877/3954747856] exists,up 146d0b0d-736c-48ea-8590-7d17db88f9e5 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:21:25.060 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:21:25.061 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 1 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 2 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 3 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 4 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 5 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 6 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 7 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 8 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: 9' 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.062 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:21:25.063 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-03-09T16:21:25.063 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-03-09T16:21:25.063 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd a/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561080 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561080 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181 5-528280977410 6-146028888133 7-167503724607 8-188978561080' 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.743 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566093 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566093 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673060 2-60129542240 3-515396075522 4-103079215181 5-528280977410 6-146028888133 7-167503724607 8-188978561080 9-450971566093' 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.0 seq 21474836591 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:21:25.744 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836591 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.1 seq 42949673060 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542240 2024-03-09T16:21:25.745 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:25.896 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cepin $ids 2024-03-09T16:21:25.896 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061' 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542241 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542241 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241' 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523' 2024-03-09T16:21:25.897 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215182 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215182 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182' 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977411 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977411 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182 5-528280977411' 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888135 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888135 2024-03-09T16:21:25.898 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182 5-528280977411 6-146028888135' 2024-03-09T16:21:25.899 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:25.899 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T16:21:26.456 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724608 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724608 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182 5-528280977411 6-146028888135 7-167503724608' 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561081 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561081 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182 5-528280977411 6-146028888135 7-167503724608 8-188978561081' 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-03-09T16:21:26.457 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673061 2-60129542241 3-515396075523 4-103079215182 5-528280977411 6-146028888135 7-167503724608 8-188978561081 9-450971566094' 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.0 seq 21474836593 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836593 2024-03-09T16:21:26.458 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-03-09T16:21:26.459 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.1 seq 42949673061 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542240 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542240 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542240' 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.2 seq 60129542240 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:21:26.675 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542240 -lt 60129542240 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075522 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075522 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075522' 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.3 seq 515396075522 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075522 -lt 515396075522 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:26.676 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215181 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215181 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215181 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215181' 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.4 seq 103079215181 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215181 -lt 103079215181 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977410 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:21:26.677 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977410 2024-03-09T16:21:26.678 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977410' 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.5 seq 528280977410 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-03-09T16:21:27.852 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888133 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888133 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888133 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888133' 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.6 seq 146028888133 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888135 -lt 146028888133 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724607 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:21:27.853 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724607 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724607 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724607' 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.7 seq 167503724607 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724608 -lt 167503724607 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561080 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561080 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561080 2024-03-09T16:21:27.854 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561080' 2024-03-09T16:21:27.855 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.8 seq 188978561080 2024-03-09T16:21:27.855 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:21:28.650 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:21:28.650 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673061 2024-03-09T16:21:28.650 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:21:28.650 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673061 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542241 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542241 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542241 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542241' 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.2 seq 60129542241 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542242 -lt 60129542241 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:28.651 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075523 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075523 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075523' 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.3 seq 515396075523 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075524 -lt 515396075523 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215182 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215182 2024-03-09T16:21:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:28.653 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215182 2024-03-09T16:21:28.653 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215182' 2024-03-09T16:21:28.653 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.4 seq 103079215182 2024-03-09T16:21:29.158 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubun88978561081 -lt 188978561080 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566093 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566093 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566093 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566093' 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: waiting osd.9 seq 450971566093 2024-03-09T16:21:29.159 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566093 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:21:29.160 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:278978: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:278978: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T16:21:29.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278977 2024-03-09T16:21:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T16:21:29.814 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215182 -lt 103079215182 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977411 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977411 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977411 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977411' 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.5 seq 528280977411 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977412 -lt 528280977411 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:29.815 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888135 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888135 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888135 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888135' 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.6 seq 146028888135 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888135 -lt 146028888135 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724608 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T16:21:29.816 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724608 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724608 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724608' 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.7 seq 167503724608 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724608 -lt 167503724608 2024-03-09T16:21:29.817 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561081 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561081 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561081 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561081' 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.8 seq 188978561081 2024-03-09T16:21:31.022 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561082 -lt 188978561081 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566094 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566094 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566094' 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: waiting osd.9 seq 450971566094 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566094 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:21:31.023 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:21:31.024 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:278980: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:278980: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T16:21:31.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T16:21:31.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T16:21:31.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:21:31.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:21:31.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:21:31.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:21:31.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:21:31.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:21:31.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:21:31.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:21:31.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:21:31.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:21:31.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:21:31.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:21:31.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:21:31.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:21:31.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:21:31.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:21:31.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:21:31.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:21:31.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:21:31.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:31.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:31.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:21:31.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:21:31.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:21:31.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:21:31.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:21:31.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:21:31.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:21:31.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:21:31.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:21:31.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:21:31.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:21:31.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:21:31.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:21:31.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:21:31.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:21:31.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:21:31.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:21:31.616 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:21:31.616 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:31.616 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:31.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:21:31.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:21:31.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:21:31.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:21:31.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:21:31.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:31.619 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:31.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:21:31.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:21:31.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:21:31.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:32.017 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:32.018 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:32.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T16:21:32.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:21:32.210 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:21:32.210 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:21:32.210 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:21:32.210 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:21:32.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:21:32.211 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:21:32.211 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:21:32.211 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:21:32.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:21:32.291 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:21:32.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:21:32.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:21:32.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T16:21:32.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:21:32.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:21:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:21:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:21:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:21:32.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:21:33.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:33.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:21:33.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:21:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-03-09T16:21:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:21:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:21:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:21:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:21:33.227 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:21:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe8dccb0-0e0c-4264-855f-27d6fcb795b3 2024-03-09T16:21:33.229 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 fe8dccb0-0e0c-4264-855f-27d6fcb795b3 2024-03-09T16:21:33.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fe8dccb0-0e0c-4264-855f-27d6fcb795b3' 2024-03-09T16:21:33.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:21:33.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNjOxlaHqBDhAABvpHsav+RxlM0pecXmwneg== 2024-03-09T16:21:33.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNjOxlaHqBDhAABvpHsav+RxlM0pecXmwneg=="}' 2024-03-09T16:21:33.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe8dccb0-0e0c-4264-855f-27d6fcb795b3 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:21:33.450 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:21:33.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:21:33.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNjOxlaHqBDhAABvpHsav+RxlM0pecXmwneg== --osd-uuid fe8dccb0-0e0c-4264-855f-27d6fcb795b3 2024-03-09T16:21:33.489 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.488+0000 7f0b53149640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:33.501 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.500+0000 7f0b53149640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:33.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.501+0000 7f0b53149640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:33.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.501+0000 7f0b53149640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:21:33.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.501+0000 7f0b53149640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:21:33.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.501+0000 7f0b53149640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:21:33.593 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:33.592+0000 7f0b53149640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:21:39.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:21:39.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:21:39.115 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:21:39.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:21:39.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:21:39.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:21:39.477 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:21:39.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:21:39.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:21:39.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:21:39.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:21:39.503 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:39.501+0000 7feb4c326640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:39.505 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:39.503+0000 7feb4c326640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:39.506 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:39.504+0000 7feb4c326640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:21:39.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:21:39.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:21:39.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:39.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:40.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:40.030 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:40.028+0000 7feb4c326640 -1 Falling back to public interface 2024-03-09T16:21:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:41.023 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:21:41.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:21:41.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:41.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:41.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:42.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:42.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:42.305 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:21:42.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:21:42.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:42.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:42.347 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:21:42.345+0000 7feb4c326640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:21:42.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:43.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:43.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:43.607 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:21:43.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:21:43.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:43.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:43.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:44.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:44.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:44.891 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:21:44.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:21:44.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:44.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:45.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:46.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:46.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:46.277 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:21:46.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:21:46.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:46.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:46.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:47.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:47.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:47.562 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:21:47.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:21:47.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:47.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:47.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:48.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:48.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:48.900 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:21:48.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:21:48.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:48.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:49.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:50.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:50.310 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:21:50.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:21:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:50.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:51.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:51.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:51.592 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:21:51.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:21:51.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:51.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:52.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:52.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:52.906 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:21:52.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:21:52.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:52.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:53.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:54.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:54.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:54.291 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:21:54.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:21:54.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:54.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:54.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:55.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:55.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:55.746 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:21:55.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:21:55.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:55.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:56.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:57.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:57.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:57.132 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:21:57.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:21:57.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:57.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:57.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:21:58.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:21:58.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:21:58.759 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:21:58.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:21:58.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:21:58.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:21:59.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:00.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:00.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:00.203 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:22:00.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:22:00.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:00.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:00.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:01.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:01.772 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:22:01.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:22:01.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:01.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:02.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:03.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:03.056 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:22:03.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:03.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:22:03.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:03.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:03.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:04.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:22:04.492 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:22:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:04.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:04.852 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2080409389,v1:127.0.0.1:6803/2080409389] [v2:127.0.0.1:6804/2080409389,v1:127.0.0.1:6805/2080409389] exists,up fe8dccb0-0e0c-4264-855f-27d6fcb795b3 2024-03-09T16:22:04.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:22:04.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:22:04.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:22:04.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:22:04.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:22:04.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:22:04.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:22:04.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:22:04.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:22:04.860 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:22:04.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4afbc39e-99e3-4485-8001-a0c73cbb52dc' 2024-03-09T16:22:04.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:22:04.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsjOxlRI8ENBAAbFl+x0CqM89OP4juAQN2mg== 2024-03-09T16:22:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsjOxlRI8ENBAAbFl+x0CqM89OP4juAQN2mg=="}' 2024-03-09T16:22:04.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4afbc39e-99e3-4485-8001-a0c73cbb52dc -i td/osd-scrub-repair/1/new.json 2024-03-09T16:22:05.202 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:22:05.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:22:05.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsjOxlRI8ENBAAbFl+x0CqM89OP4juAQN2mg== --osd-uuid 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:22:05.241 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.240+0000 7f4fb2a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:05.285 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.284+0000 7f4fb2a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:05.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.285+0000 7f4fb2a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:05.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.286+0000 7f4fb2a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:22:05.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.286+0000 7f4fb2a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:22:05.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.286+0000 7f4fb2a0e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:22:05.351 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:05.350+0000 7f4fb2a0e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:22:11.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:22:11.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:22:11.500 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:22:11.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:22:11.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:22:11.853 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:22:11.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:22:11.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:22:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:22:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:22:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:22:11.879 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:11.878+0000 7f475fa0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:11.881 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:11.880+0000 7f475fa0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:11.882 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:11.881+0000 7f475fa0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:12.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:22:12.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:22:12.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:22:12.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:22:12.126 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:22:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:22:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:22:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:12.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:12.407 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:12.407+0000 7f475fa0c640 -1 Falling back to public interface 2024-03-09T16:22:13.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:13.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:22:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:22:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:13.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:14.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:14.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:14.672 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:22:14.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:22:14.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:14.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:14.765 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:14.764+0000 7f475fa0c640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:22:14.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:15.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:15.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:15.983 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:22:15.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:22:15.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:15.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:16.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:17.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:17.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:17.266 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:22:17.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:22:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:17.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:17.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:18.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:18.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:18.703 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:22:18.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:22:18.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:18.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:18.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:19.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:19.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:19.984 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:22:19.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:22:19.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:19.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:20.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:21.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:21.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:21.263 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:22:21.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:22:21.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:21.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:21.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:22.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:22.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:22.550 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:22:22.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:22:22.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:22.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:22.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:23.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:23.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:23.831 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:22:23.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:22:23.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:23.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:24.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:25.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:25.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:25.110 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:22:25.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:22:25.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:25.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:25.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:26.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:26.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:26.396 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:22:26.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:22:26.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:26.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:27.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:28.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:28.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:28.006 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:22:28.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:22:28.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:28.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:28.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:29.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:29.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:29.310 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:22:29.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:22:29.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:29.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:29.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:30.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:30.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:30.784 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:22:30.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:22:30.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:30.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:31.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:32.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:32.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:32.357 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:22:32.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:22:32.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:32.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:32.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:33.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:33.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:33.641 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:22:33.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:22:33.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:33.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:34.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:35.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:35.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:35.165 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:22:35.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:22:35.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:35.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:35.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:36.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:36.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:36.667 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:22:36.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:22:36.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:36.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:37.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:38.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:38.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:38.147 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:22:38.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:22:38.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:38.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:22:38.425 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1769403606,v1:127.0.0.1:6811/1769403606] [v2:127.0.0.1:6812/1769403606,v1:127.0.0.1:6813/1769403606] exists,up 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:22:38.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:22:38.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:22:38.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:22:38.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-03-09T16:22:38.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:22:38.686 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:22:38.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:22:38.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:22:39.069 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:22:39.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:22:40.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:22:42.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-03-09T16:22:42.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:22:42.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:22:42.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:22:42.036 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:22:42.037 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:22:42.037 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:22:42.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:22:42.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:22:42.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:22:42.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:22:42.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:22:42.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:22:42.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:22:42.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:22:42.134 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:22:42.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:22:42.402 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:22:42.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:22:42.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:22:42.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:22:42.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-09T16:22:42.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-09T16:22:42.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-09T16:22:42.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:22:42.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672969' 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-09T16:22:42.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:22:42.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:22:42.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-09T16:22:42.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:22:42.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-09T16:22:42.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-09T16:22:42.614 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836495 2024-03-09T16:22:42.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:22:42.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-09T16:22:42.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:22:43.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:22:43.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:22:44.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-09T16:22:44.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:22:44.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-09T16:22:44.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:22:44.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:22:44.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-09T16:22:44.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:22:44.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-09T16:22:44.168 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672969 2024-03-09T16:22:44.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-09T16:22:44.169 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:22:44.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-03-09T16:22:44.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:22:44.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:22:44.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:22:44.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:22:44.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:22:44.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:22:44.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:22:44.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:22:44.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:22:44.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:22:44.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:22:45.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:22:45.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:22:45.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:22:45.043 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:22:45.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:22:45.803 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:22:45.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:22:46.372 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:22:46.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:22:46.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:22:46.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T16:22:46.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-03-09T16:22:46.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-09T16:22:46.618 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T16:22:46.618 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-09T16:22:46.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T16:22:46.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T16:22:46.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T16:22:46.619 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:22:46.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T16:22:46.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T16:22:46.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T16:22:47.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-09T16:22:47.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:22:47.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:22:47.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:22:47.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:22:47.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:22:47.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:22:47.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:22:47.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T16:22:47.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:22:47.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:22:47.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:22:47.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:22:47.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:22:47.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T16:22:49.003 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T16:22:49.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:22:49.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:22:49.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:22:49.521 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:22:49.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:22:49.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:22:49.523 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:22:49.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:22:49.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:22:49.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:22:49.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:22:49.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:22:49.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:22:49.552 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:49.548+0000 7f6fe0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:49.563 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:49.561+0000 7f6fe0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:49.564 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:49.562+0000 7f6fe0c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:49.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:50.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:50.083 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:50.081+0000 7f6fe0c0b640 -1 Falling back to public interface 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:51.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:52.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:52.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:52.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:22:52.359 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:22:52.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:52.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:52.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:22:52.434+0000 7f6fe0c0b640 -1 osd.0 21 log_to_monitors true 2024-03-09T16:22:52.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:22:53.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4047516162,v1:127.0.0.1:6803/4047516162] [v2:127.0.0.1:6804/4047516162,v1:127.0.0.1:6805/4047516162] exists,up fe8dccb0-0e0c-4264-855f-27d6fcb795b3 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:22:54.078 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:22:54.079 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:22:54.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:22:54.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:22:54.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:22:54.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:22:54.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:22:54.433 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:22:54.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:22:54.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:22:54.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:22:54.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-09T16:22:54.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-09T16:22:54.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-09T16:22:54.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:22:54.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:22:54.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T16:22:54.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T16:22:54.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672973' 2024-03-09T16:22:54.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:22:54.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-09T16:22:54.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:22:54.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:22:54.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-09T16:22:54.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:22:54.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-09T16:22:54.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-09T16:22:54.645 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149698 2024-03-09T16:22:54.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:22:54.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-09T16:22:54.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:22:55.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:22:55.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:22:56.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-09T16:22:56.193 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:22:56.193 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T16:22:56.193 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:22:56.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:22:56.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T16:22:56.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:22:56.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T16:22:56.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T16:22:56.196 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672973 2024-03-09T16:22:56.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:22:56.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T16:22:56.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:22:56.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:22:56.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:22:56.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:22:56.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:22:56.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:22:57.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:22:57.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:22:57.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:22:57.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T16:22:57.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:22:57.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:22:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:22:57.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:22:39.014537+0000 2024-03-09T16:22:57.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T16:22:58.224 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T16:22:58.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T16:22:39.014537+0000 2024-03-09T16:22:58.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T16:22:58.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:22:39.014537+0000 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:22:58.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:22:58.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:22:58.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:22:58.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:22:59.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:22:59.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:22:59.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:22:59.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:22:59.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:22:59.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:22:59.507 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:22:59.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:22:59.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:00.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:00.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:00.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:00.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:00.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:01.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:23:01.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:02.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:02.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:02.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:02.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:02.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:02.066 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:02.066 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:02.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:23:02.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:03.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:03.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:03.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:03.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:03.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:03.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:03.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:03.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:23:03.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:04.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:04.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:04.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:04.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:04.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:04.627 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:04.627 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:39.014537+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:23:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:05.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:05.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:05.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:59.037661+0000 '>' 2024-03-09T16:22:39.014537+0000 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:23:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:23:06.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:06.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T16:23:07.723 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T16:23:07.723 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:23:07.723 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:23:07.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:23:07.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:23:07.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:23:07.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:23:07.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:07.995 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:23:07.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:23:07.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:23:07.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:23:07.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:23:07.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:23:07.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:23:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:08.019+0000 7f9305a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:08.021+0000 7f9305a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:08.023 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:08.022+0000 7f9305a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:23:08.273 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:23:08.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:08.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:08.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:08.548 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:08.548+0000 7f9305a0e640 -1 Falling back to public interface 2024-03-09T16:23:09.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:09.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:09.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:23:09.545 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:23:09.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:09.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:09.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:10.744 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:10.743+0000 7f9305a0e640 -1 osd.1 28 log_to_monitors true 2024-03-09T16:23:10.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:10.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:10.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:23:10.825 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:23:10.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:10.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:11.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:12.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:12.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:12.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:23:12.128 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:23:12.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:12.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:12.424 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2561540535,v1:127.0.0.1:6811/2561540535] [v2:127.0.0.1:6812/2561540535,v1:127.0.0.1:6813/2561540535] exists,up 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:23:12.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:23:12.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:23:12.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:23:12.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:23:12.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:23:12.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:23:12.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:23:12.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:23:12.779 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:23:12.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:23:12.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:12.779 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:23:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149703 2024-03-09T16:23:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149703 2024-03-09T16:23:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703' 2024-03-09T16:23:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:12.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:23:12.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-09T16:23:12.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-09T16:23:12.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703 1-137438953474' 2024-03-09T16:23:12.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:12.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149703 2024-03-09T16:23:12.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:12.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:23:12.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149703 2024-03-09T16:23:12.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:12.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149703 2024-03-09T16:23:12.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2024-03-09T16:23:12.989 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149703 2024-03-09T16:23:12.989 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:13.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149703 2024-03-09T16:23:13.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:23:14.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:23:14.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:14.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149703 2024-03-09T16:23:14.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:14.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-09T16:23:14.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:14.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:23:14.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-09T16:23:14.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:14.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-09T16:23:14.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-09T16:23:14.539 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 137438953474 2024-03-09T16:23:14.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:23:14.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-09T16:23:14.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:23:14.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:14.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:23:15.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:23:15.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:23:15.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:23:15.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:15.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:23:15.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:23:15.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:23:15.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:23:15.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:23:15.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-03-09T16:23:15.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T16:23:15.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T16:23:15.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T16:23:15.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:23:16.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:23:16.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T16:23:17.735 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T16:23:18.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:23:18.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:23:18.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:23:18.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:23:18.256 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:23:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:23:18.283 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:18.282+0000 7ffa70813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:18.285 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:18.284+0000 7ffa70813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:18.287 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:18.286+0000 7ffa70813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:18.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:23:18.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:23:18.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:23:18.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:18.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:18.807 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:18.806+0000 7ffa70813640 -1 Falling back to public interface 2024-03-09T16:23:18.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:19.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:20.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:21.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:21.135 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:21.134+0000 7ffa70813640 -1 osd.1 34 log_to_monitors true 2024-03-09T16:23:21.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:22.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/4144890995,v1:127.0.0.1:6811/4144890995] [v2:127.0.0.1:6812/4144890995,v1:127.0.0.1:6813/4144890995] exists,up 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:23:22.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:23:22.646 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:23:22.646 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:23:22.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:23:22.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:23:22.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:23:22.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:23:23.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:23:23.000 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:23:23.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:23:23.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:23.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:23:23.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149706 2024-03-09T16:23:23.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149706 2024-03-09T16:23:23.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149706' 2024-03-09T16:23:23.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:23.103 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:23:23.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-03-09T16:23:23.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-03-09T16:23:23.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149706 1-163208757250' 2024-03-09T16:23:23.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:23.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149706 2024-03-09T16:23:23.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:23.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:23:23.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149706 2024-03-09T16:23:23.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:23.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149706 2024-03-09T16:23:23.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149706' 2024-03-09T16:23:23.210 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149706 2024-03-09T16:23:23.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:23.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149706 -lt 111669149706 2024-03-09T16:23:23.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:23.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-03-09T16:23:23.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:23.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:23:23.501 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-03-09T16:23:23.501 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-03-09T16:23:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-03-09T16:23:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 163208757250 2024-03-09T16:23:23.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:23:23.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-03-09T16:23:23.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:23:23.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:23.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:24.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:23:24.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:23:24.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:23:24.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:23:24.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:24.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:24.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T16:23:24.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:24.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:24.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:25.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:25.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T16:23:25.536 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:25.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:25.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:25.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:25.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:25.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:25.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:59.037661+0000 '>' 2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:25.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:26.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:26.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:26.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:26.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:26.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:26.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:26.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:27.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:22:59.037661+0000 '>' 2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:27.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:23:28.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:23:28.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:23:28.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T16:23:28.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T16:23:28.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:23:28.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:23:28.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:23:26.034469+0000 '>' 2024-03-09T16:22:59.037661+0000 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:23:28.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:23:28.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:28.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T16:23:30.045 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T16:23:30.046 INFO:tasks.workunit.client.0.smithi146.stdout:_ 2024-03-09T16:23:30.046 INFO:tasks.workunit.client.0.smithi146.stdout:snapset 2024-03-09T16:23:30.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:23:30.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:23:30.319 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:23:30.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:23:30.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:23:30.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:23:30.321 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:23:30.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:23:30.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:23:30.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:23:30.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:23:30.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:23:30.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:23:30.348 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:30.347+0000 7f7767a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:30.350 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:30.349+0000 7f7767a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:30.351 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:30.350+0000 7f7767a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:30.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:30.873 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:30.872+0000 7f7767a11640 -1 Falling back to public interface 2024-03-09T16:23:30.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:31.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:32.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:33.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:33.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:33.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:23:33.155 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:23:33.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:33.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:33.161 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:33.161+0000 7f7767a11640 -1 osd.1 39 log_to_monitors true 2024-03-09T16:23:33.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:34.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/2009692569,v1:127.0.0.1:6811/2009692569] [v2:127.0.0.1:6812/2009692569,v1:127.0.0.1:6813/2009692569] exists,up 4afbc39e-99e3-4485-8001-a0c73cbb52dc 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:23:34.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:23:34.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:23:34.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:23:34.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:23:34.791 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:23:34.791 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:23:34.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:23:34.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:23:34.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:23:34.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:23:35.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:23:35.148 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:23:35.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:23:35.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:35.148 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:23:35.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149709 2024-03-09T16:23:35.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149709 2024-03-09T16:23:35.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709' 2024-03-09T16:23:35.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:23:35.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:23:35.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-09T16:23:35.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-09T16:23:35.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709 1-184683593730' 2024-03-09T16:23:35.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:35.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149709 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149709 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149709 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149709' 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 111669149709 2024-03-09T16:23:35.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:35.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149708 -lt 111669149709 2024-03-09T16:23:35.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:23:36.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:23:36.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149708 -lt 111669149709 2024-03-09T16:23:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:23:37.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:23:37.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:23:38.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149709 -lt 111669149709 2024-03-09T16:23:38.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:23:38.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-03-09T16:23:38.194 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:23:38.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:23:38.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-03-09T16:23:38.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:23:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-09T16:23:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-03-09T16:23:38.197 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 184683593730 2024-03-09T16:23:38.197 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:23:38.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-09T16:23:38.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:23:38.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:38.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:23:38.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:23:38.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:23:39.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:23:39.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:23:39.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:23:39.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:23:39.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:23:39.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:23:39.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:23:39.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T16:23:39.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T16:23:39.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:23:39.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:23:39.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:23:39.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:23:39.458 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:23:39.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:23:39.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:23:39.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:23:39.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:23:39.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:23:39.565 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:23:39.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:23:39.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:23:39.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:23:39.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:23:39.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:23:39.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:23:39.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:23:39.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:23:39.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:23:39.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:23:39.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:23:39.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:23:39.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:23:39.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:39.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:39.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:23:39.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:23:39.587 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:23:39.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:23:39.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:23:39.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:23:39.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:23:39.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:23:39.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:23:39.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:23:39.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:23:39.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:23:39.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:23:39.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:23:39.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:23:39.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:23:39.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:39.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:39.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:23:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:23:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:23:39.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:23:39.596 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:23:39.596 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:39.596 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:39.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-09T16:23:39.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:23:39.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:23:39.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:23:39.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:39.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:39.989 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:39.989 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:39.989 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:39.989 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:39.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:23:40.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:23:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:23:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:40.183 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:23:40.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:23:40.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:23:40.265 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:23:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:23:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:23:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:40.266 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:23:40.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:23:40.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:23:40.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-09T16:23:40.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:23:40.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:23:40.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:23:40.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:23:40.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:23:40.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:23:41.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:41.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:23:41.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:23:41.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-09T16:23:41.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:23:41.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-09T16:23:41.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:23:41.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:23:41.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:23:41.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:23:41.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:23:41.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:23:41.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:23:41.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:23:41.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7b665b6-d518-4347-b205-96d56d660493' 2024-03-09T16:23:41.192 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:23:41.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:23:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANjexlSBBFDBAAxBIYKjwDi2uwZvzHEVy/Hg== 2024-03-09T16:23:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANjexlSBBFDBAAxBIYKjwDi2uwZvzHEVy/Hg=="}' 2024-03-09T16:23:41.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7b665b6-d518-4347-b205-96d56d660493 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:23:41.413 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:23:41.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:23:41.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANjexlSBBFDBAAxBIYKjwDi2uwZvzHEVy/Hg== --osd-uuid e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:23:41.451 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.450+0000 7f66df61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:41.463 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.463+0000 7f66df61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:41.464 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.464+0000 7f66df61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:41.465 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.464+0000 7f66df61c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:23:41.465 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.464+0000 7f66df61c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:23:41.465 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.464+0000 7f66df61c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:23:41.547 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:41.546+0000 7f66df61c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:23:47.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:23:47.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:23:47.094 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:23:47.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:23:47.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:23:47.457 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:23:47.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:23:47.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:23:47.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:23:47.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:23:47.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:23:47.483 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:47.482+0000 7f83aa60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:47.510 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:47.509+0000 7f83aa60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:47.511 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:47.510+0000 7f83aa60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:23:47.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:47.773 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:23:47.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:23:47.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:47.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:48.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:48.046 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:48.046+0000 7f83aa60f640 -1 Falling back to public interface 2024-03-09T16:23:49.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:49.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:49.045 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:23:49.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:23:49.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:49.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:49.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:50.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:50.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:50.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:23:50.321 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:23:50.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:50.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:50.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:23:50.367+0000 7f83aa60f640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:23:50.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:51.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:51.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:51.636 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:23:51.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:23:51.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:51.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:51.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:52.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:52.921 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:23:52.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:52.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:23:52.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:52.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:53.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:54.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:54.298 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:23:54.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:54.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:23:54.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:54.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:54.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:55.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:55.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:55.586 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:23:55.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:23:55.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:55.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:55.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:56.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:56.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:56.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:23:56.930 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:23:56.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:56.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:57.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:58.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:58.323 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:23:58.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:58.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:23:58.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:58.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:58.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:23:59.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:23:59.606 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:23:59.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:23:59.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:23:59.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:23:59.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:23:59.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:00.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:00.977 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:24:00.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:00.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:24:00.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:00.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:01.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:02.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:02.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:02.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:24:02.336 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:24:02.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:02.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:02.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:03.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:03.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:03.800 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:24:03.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:24:03.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:03.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:04.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:05.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:05.629 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:24:05.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:05.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:24:05.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:05.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:06.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:07.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:07.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:07.034 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:24:07.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:24:07.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:07.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:07.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:08.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:08.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:08.350 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:24:08.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:24:08.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:08.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:08.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:09.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:09.875 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:24:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:24:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:09.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:10.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:11.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:11.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:11.249 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:24:11.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:24:11.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:11.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:12.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:13.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:13.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:13.097 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:24:13.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:24:13.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:13.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:24:13.379 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1799342079,v1:127.0.0.1:6803/1799342079] [v2:127.0.0.1:6804/1799342079,v1:127.0.0.1:6805/1799342079] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:24:13.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:24:13.384 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:24:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:24:13.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:24:13.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:24:13.386 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:24:13.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ac32d04f-e700-4416-bcb1-3a5636eea15d' 2024-03-09T16:24:13.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:24:13.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtjexlD2TIFxAAgg3ZnXZJRvr5Vqm0ETukxQ== 2024-03-09T16:24:13.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtjexlD2TIFxAAgg3ZnXZJRvr5Vqm0ETukxQ=="}' 2024-03-09T16:24:13.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac32d04f-e700-4416-bcb1-3a5636eea15d -i td/osd-scrub-repair/1/new.json 2024-03-09T16:24:13.773 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:24:13.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:24:13.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtjexlD2TIFxAAgg3ZnXZJRvr5Vqm0ETukxQ== --osd-uuid ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:24:13.812 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.810+0000 7f2a3f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:13.855 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.854+0000 7f2a3f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:13.856 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.855+0000 7f2a3f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:13.857 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.856+0000 7f2a3f40d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:24:13.857 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.856+0000 7f2a3f40d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:24:13.857 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.856+0000 7f2a3f40d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:24:13.930 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:13.929+0000 7f2a3f40d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:24:19.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:24:19.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:24:19.982 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:24:19.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:24:19.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:24:20.358 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:24:20.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:24:20.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:24:20.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:24:20.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:24:20.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:24:20.384 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:20.383+0000 7f0d0920e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:20.386 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:20.385+0000 7f0d0920e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:20.387 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:20.387+0000 7f0d0920e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:20.633 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:24:20.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:24:20.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:20.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:20.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:20.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:20.912+0000 7f0d0920e640 -1 Falling back to public interface 2024-03-09T16:24:21.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:21.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:21.902 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:24:21.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:24:21.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:21.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:22.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:23.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:23.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:23.178 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:24:23.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:24:23.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:23.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:23.235 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:23.234+0000 7f0d0920e640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:24:23.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:24.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:24.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:24.520 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:24:24.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:24:24.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:24.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:24.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:25.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:25.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:25.826 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:24:25.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:24:25.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:25.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:26.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:27.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:27.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:27.196 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:24:27.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:24:27.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:27.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:27.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:28.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:28.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:28.512 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:24:28.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:24:28.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:28.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:28.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:29.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:29.798 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:24:29.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:29.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:24:29.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:29.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:30.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:31.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:31.077 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:24:31.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:31.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:24:31.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:31.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:31.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:32.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:32.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:32.358 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:24:32.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:24:32.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:32.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:32.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:33.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:33.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:33.706 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:24:33.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:24:33.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:33.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:34.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:35.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:35.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:35.002 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:24:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:24:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:35.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:35.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:36.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:36.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:36.397 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:24:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:24:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:36.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:37.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:38.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:38.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:38.049 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:24:38.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:24:38.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:38.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:38.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:39.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:39.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:39.718 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:24:39.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:24:39.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:39.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:40.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:41.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:41.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:41.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:24:41.429 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:24:41.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:41.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:41.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:42.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:42.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:42.980 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:24:42.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:24:42.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:42.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:44.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:45.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:45.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:45.060 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:24:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:24:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:45.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:46.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:46.470 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:24:46.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:46.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:24:46.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:46.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:24:46.744 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1618311849,v1:127.0.0.1:6811/1618311849] [v2:127.0.0.1:6812/1618311849,v1:127.0.0.1:6813/1618311849] exists,up ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:24:46.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:24:46.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:24:46.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:24:46.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:24:46.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:24:46.752 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:24:46.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 88e8cf70-5022-43db-8499-6d7479e03437' 2024-03-09T16:24:46.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:24:46.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOjexlRrSULRAA0bmvrjl/vQ9lyULNYL+PgQ== 2024-03-09T16:24:46.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOjexlRrSULRAA0bmvrjl/vQ9lyULNYL+PgQ=="}' 2024-03-09T16:24:46.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88e8cf70-5022-43db-8499-6d7479e03437 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:24:47.135 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:24:47.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:24:47.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOjexlRrSULRAA0bmvrjl/vQ9lyULNYL+PgQ== --osd-uuid 88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:24:47.175 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.174+0000 7f06ce815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:47.219 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.218+0000 7f06ce815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:47.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.219+0000 7f06ce815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:47.220 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.220+0000 7f06ce815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:24:47.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.220+0000 7f06ce815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:24:47.221 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.220+0000 7f06ce815640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:24:47.286 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:47.285+0000 7f06ce815640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:24:53.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:24:53.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:24:53.325 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:24:53.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:24:53.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:24:53.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:24:53.678 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:24:53.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:24:53.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:24:53.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:24:53.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:24:53.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:53.703+0000 7f618a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:53.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:53.705+0000 7f618a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:53.706 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:53.706+0000 7f618a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:24:53.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:24:53.950 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:24:53.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:53.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:24:54.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:54.223 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:54.223+0000 7f618a20c640 -1 Falling back to public interface 2024-03-09T16:24:55.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:55.222 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:24:55.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:55.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:24:55.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:55.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:24:55.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:56.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:56.508 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:24:56.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:56.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:24:56.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:56.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:24:56.649 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:24:56.648+0000 7f618a20c640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:24:56.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:57.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:57.842 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:24:57.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:57.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:24:57.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:57.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:24:58.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:24:59.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:24:59.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:24:59.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:24:59.123 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:24:59.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:24:59.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:24:59.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:00.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:00.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:00.536 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:25:00.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:25:00.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:00.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:00.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:01.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:01.848 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:25:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:25:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:01.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:02.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:03.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:03.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:03.132 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:25:03.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:25:03.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:03.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:03.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:04.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:04.508 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:25:04.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:04.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:25:04.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:04.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:04.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:05.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:05.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:05.840 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:25:05.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:25:05.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:05.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:06.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:07.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:07.122 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:25:07.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:07.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:25:07.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:07.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:07.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:08.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:08.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:08.482 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:25:08.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:25:08.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:08.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:08.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:09.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:09.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:09.768 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:25:09.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:25:09.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:09.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:10.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:11.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:11.164 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:25:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:25:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:11.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:11.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:12.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:12.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:12.523 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:25:12.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:25:12.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:12.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:13.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:14.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:14.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:14.311 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:25:14.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:25:14.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:14.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:14.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:15.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:15.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:15.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:25:15.778 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:25:15.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:15.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:16.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:17.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:17.378 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:25:17.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:17.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:25:17.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:17.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:18.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:19.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:19.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:19.165 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:25:19.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:25:19.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:19.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:19.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:20.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:20.651 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:25:20.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:20.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:25:20.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:20.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:25:21.000 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3324632197,v1:127.0.0.1:6819/3324632197] [v2:127.0.0.1:6820/3324632197,v1:127.0.0.1:6821/3324632197] exists,up 88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:25:21.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:25:21.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:25:21.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:25:21.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-09T16:25:21.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:25:21.285 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:25:21.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:25:21.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:25:21.672 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:25:21.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:25:22.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:25:24.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-09T16:25:24.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T16:25:25.271 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'foo' created 2024-03-09T16:25:25.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:25:26.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-03-09T16:25:26.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:25:26.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:25:26.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T16:25:26.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:25:26.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-09T16:25:26.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:25:26.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:25:27.439 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:25:27.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:25:28.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:25:28.453 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:25:28.454 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:25:28.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:25:28.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:25:28.454 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:25:28.551 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:25:28.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:25:28.848 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:28.848 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:25:28.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:25:28.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:28.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:25:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T16:25:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T16:25:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T16:25:28.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:28.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:25:29.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-09T16:25:29.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-09T16:25:29.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672974' 2024-03-09T16:25:29.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:29.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:25:29.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T16:25:29.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T16:25:29.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672974 2-64424509446' 2024-03-09T16:25:29.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:29.169 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T16:25:29.169 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:29.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:25:29.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T16:25:29.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:29.171 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836498 2024-03-09T16:25:29.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T16:25:29.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T16:25:29.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:29.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T16:25:29.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:29.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-09T16:25:29.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:29.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:25:29.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-09T16:25:29.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:29.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-09T16:25:29.448 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672974 2024-03-09T16:25:29.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-09T16:25:29.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:25:29.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-09T16:25:29.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:29.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T16:25:29.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:29.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:25:29.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T16:25:29.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:29.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T16:25:29.754 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509446 2024-03-09T16:25:29.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T16:25:29.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:25:30.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-09T16:25:30.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:25:30.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:30.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:30.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:25:30.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:25:30.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:25:30.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:25:30.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:30.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:30.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:25:30.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:25:30.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:25:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:25:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:25:31.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:25:31.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:25:31.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:25:31.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:25:31.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:25:31.356 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:31.356 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:25:31.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:25:31.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:31.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:25:31.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-09T16:25:31.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-09T16:25:31.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-09T16:25:31.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:31.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:25:31.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T16:25:31.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T16:25:31.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672975' 2024-03-09T16:25:31.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:31.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:25:31.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-09T16:25:31.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-09T16:25:31.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672975 2-64424509448' 2024-03-09T16:25:31.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:31.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-09T16:25:31.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:31.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:25:31.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-09T16:25:31.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-09T16:25:31.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-09T16:25:31.772 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836499 2024-03-09T16:25:31.772 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:32.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-09T16:25:32.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:25:33.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:25:33.044 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:33.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-03-09T16:25:33.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:33.328 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T16:25:33.328 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:33.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:25:33.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T16:25:33.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:33.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T16:25:33.331 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672975 2024-03-09T16:25:33.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T16:25:33.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:25:33.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-03-09T16:25:33.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:33.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-09T16:25:33.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:33.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:25:33.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-09T16:25:33.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:33.625 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509448 2024-03-09T16:25:33.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-09T16:25:33.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-09T16:25:33.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:25:33.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-09T16:25:33.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:25:33.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:33.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:34.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:25:34.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:25:34.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:25:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:25:34.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:25:34.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:34.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:34.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:25:34.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:25:34.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:25:34.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:25:34.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:25:34.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:25:35.249 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:25:35.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:25:35.767 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:25:35.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:25:35.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:25:35.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:25:36.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:25:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:25:36.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:25:36.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:25:36.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:36.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:25:36.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:25:36.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:25:36.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:25:36.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:25:36.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:25:38.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:25:38.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:25:38.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:25:38.635 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:25:38.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:25:38.636 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:25:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:25:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:25:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:25:38.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:25:38.661 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:38.660+0000 7f34453f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:38.663 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:38.662+0000 7f34453f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:38.664 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:38.664+0000 7f34453f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:38.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:25:39.182 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:39.181+0000 7f34453f8640 -1 Falling back to public interface 2024-03-09T16:25:39.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:40.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:40.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:40.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:25:40.190 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:40.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:40.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:25:40.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:41.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:25:41.674 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:41.673+0000 7f34453f8640 -1 osd.1 35 log_to_monitors true 2024-03-09T16:25:41.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:42.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:42.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:42.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:25:42.773 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:25:42.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:42.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:25:43.057 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3478207539,v1:127.0.0.1:6811/3478207539] [v2:127.0.0.1:6812/3478207539,v1:127.0.0.1:6813/3478207539] exists,up ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:25:43.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:25:43.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:25:43.060 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:25:43.144 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:43.559 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:25:43.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T16:25:43.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T16:25:43.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T16:25:43.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:43.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:25:43.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T16:25:43.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T16:25:43.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-171798691842' 2024-03-09T16:25:43.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:43.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:25:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-09T16:25:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-09T16:25:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-171798691842 2-64424509451' 2024-03-09T16:25:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:43.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T16:25:43.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:43.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:25:43.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T16:25:43.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:43.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T16:25:43.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T16:25:43.879 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836503 2024-03-09T16:25:43.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:44.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-09T16:25:44.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:25:45.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:25:45.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:45.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T16:25:45.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:45.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-09T16:25:45.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:45.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:25:45.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-09T16:25:45.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:45.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T16:25:45.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-09T16:25:45.447 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 171798691842 2024-03-09T16:25:45.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:25:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T16:25:45.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:45.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-09T16:25:45.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:45.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:25:45.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-09T16:25:45.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:45.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-09T16:25:45.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-09T16:25:45.722 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509451 2024-03-09T16:25:45.722 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:25:45.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-03-09T16:25:45.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:25:45.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:45.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:25:46.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:25:46.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:25:46.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:25:46.627 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:25:46.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:46.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:25:46.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:25:47.471 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:25:47.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:25:47.855 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:25:47.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:25:47.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:25:47.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:25:48.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-09T16:25:48.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:25:48.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:25:48.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-09T16:25:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-03-09T16:25:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-03-09T16:25:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1888e-05 s, 33.1 MB/s 2024-03-09T16:25:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:48.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:25:48.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:25:49.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:25:49.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:25:49.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:25:51.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:25:51.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:25:51.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:25:51.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:25:51.143 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:51.142+0000 7f0997c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:51.145 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:51.144+0000 7f0997c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:51.146 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:51.145+0000 7f0997c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:25:51.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:25:51.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:25:51.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:25:51.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:25:51.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:25:51.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:51.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:25:51.396 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:25:51.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:51.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:25:51.663 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:51.662+0000 7f0997c14640 -1 Falling back to public interface 2024-03-09T16:25:51.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:52.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:25:52.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:53.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:53.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:53.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:25:53.978 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:25:53.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:53.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:25:54.013 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:25:54.012+0000 7f0997c14640 -1 osd.0 44 log_to_monitors true 2024-03-09T16:25:54.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:25:55.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:25:55.630 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/378700731,v1:127.0.0.1:6803/378700731] [v2:127.0.0.1:6804/378700731,v1:127.0.0.1:6805/378700731] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:25:55.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:25:55.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:25:55.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:25:55.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:25:55.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:25:55.632 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:25:55.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:25:55.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:25:55.987 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:25:55.988 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:25:55.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:25:55.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:55.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:25:56.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-09T16:25:56.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-09T16:25:56.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-03-09T16:25:56.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:56.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:25:56.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T16:25:56.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T16:25:56.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845' 2024-03-09T16:25:56.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:25:56.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:25:56.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T16:25:56.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T16:25:56.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845 2-64424509455' 2024-03-09T16:25:56.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:56.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-03-09T16:25:56.329 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:56.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:25:56.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-03-09T16:25:56.330 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:56.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-09T16:25:56.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-03-09T16:25:56.332 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 210453397506 2024-03-09T16:25:56.332 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:56.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397506 2024-03-09T16:25:56.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:25:57.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:25:57.604 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:25:57.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-03-09T16:25:57.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:57.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-09T16:25:57.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:57.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:25:57.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-09T16:25:57.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:57.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T16:25:57.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-09T16:25:57.890 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 171798691845 2024-03-09T16:25:57.890 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:25:58.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691845 2024-03-09T16:25:58.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:25:58.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T16:25:58.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:25:58.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:25:58.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T16:25:58.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:25:58.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T16:25:58.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T16:25:58.166 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509455 2024-03-09T16:25:58.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:25:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T16:25:58.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:25:58.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:58.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:25:58.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:25:59.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:25:59.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:25:59.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:25:59.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:25:59.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:25:59.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:25:59.826 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:25:59.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:26:00.344 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:26:00.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:26:00.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:26:00.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:26:00.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:26:00.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:26:00.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:26:00.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:26:00.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:26:00.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:26:00.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:26:00.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:26:00.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:26:00.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:26:00.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:26:00.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:26:00.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:26:00.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:26:00.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-09T16:26:02.833 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:26:03.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:26:03.356 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:26:03.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:26:03.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:26:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:03.380+0000 7ff3466c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:03.382+0000 7ff3466c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:03.384+0000 7ff3466c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:03.899 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:03.898+0000 7ff3466c2640 -1 Falling back to public interface 2024-03-09T16:26:03.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:04.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:05.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:06.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:06.201 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:06.200+0000 7ff3466c2640 -1 osd.1 53 log_to_monitors true 2024-03-09T16:26:06.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:07.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:07.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:07.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:26:07.490 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:26:07.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:07.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:07.850 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/3165260814,v1:127.0.0.1:6811/3165260814] [v2:127.0.0.1:6812/3165260814,v1:127.0.0.1:6813/3165260814] exists,up ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:26:07.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:26:07.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:26:07.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:26:07.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:26:07.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:26:07.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:26:07.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:26:07.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:26:07.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:26:07.936 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:26:08.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:26:08.305 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:08.305 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:26:08.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:26:08.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:08.305 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:26:08.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-03-09T16:26:08.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-03-09T16:26:08.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509' 2024-03-09T16:26:08.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:08.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:26:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-03-09T16:26:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-03-09T16:26:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170' 2024-03-09T16:26:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:08.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:26:08.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T16:26:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T16:26:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170 2-64424509458' 2024-03-09T16:26:08.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397509 2024-03-09T16:26:08.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:08.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:26:08.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397509 2024-03-09T16:26:08.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:08.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-03-09T16:26:08.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397509' 2024-03-09T16:26:08.644 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 210453397509 2024-03-09T16:26:08.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:08.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397509 2024-03-09T16:26:08.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:26:09.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:26:09.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397509 -lt 210453397509 2024-03-09T16:26:10.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:10.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-03-09T16:26:10.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:10.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:26:10.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-03-09T16:26:10.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-03-09T16:26:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-03-09T16:26:10.201 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103170 2024-03-09T16:26:10.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:26:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-03-09T16:26:10.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:10.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T16:26:10.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:10.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:26:10.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T16:26:10.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T16:26:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T16:26:10.477 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509458 2024-03-09T16:26:10.478 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:26:10.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-09T16:26:10.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:26:10.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:10.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:11.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:26:11.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:11.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:11.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:26:11.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:11.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:11.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-09T16:26:11.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:26:11.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:26:12.076 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:26:12.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:26:12.603 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:26:12.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:26:12.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:26:12.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:26:12.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-09T16:26:12.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:26:12.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:26:12.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-09T16:26:13.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:26:13.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:26:13.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:26:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:26:15.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:26:15.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:26:15.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:26:15.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:26:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:26:15.752 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:26:15.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:26:15.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:26:15.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:26:15.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:26:15.774 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:15.771+0000 7f697eeaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:15.775 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:15.773+0000 7f697eeaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:15.777 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:15.775+0000 7f697eeaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:16.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:26:16.029 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:26:16.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:16.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:26:16.293 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:16.291+0000 7f697eeaa640 -1 Falling back to public interface 2024-03-09T16:26:16.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:17.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:26:17.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:18.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:18.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:18.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:26:18.618 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:26:18.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:18.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:26:18.713 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:18.710+0000 7f697eeaa640 -1 osd.0 62 log_to_monitors true 2024-03-09T16:26:18.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:19.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:19.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:19.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:26:19.912 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:26:19.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:19.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:26:20.287 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/3124547966,v1:127.0.0.1:6803/3124547966] [v2:127.0.0.1:6804/3124547966,v1:127.0.0.1:6805/3124547966] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:26:20.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:26:20.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:26:20.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:26:20.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:26:20.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:26:20.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:26:20.757 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:20.757 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:26:20.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:26:20.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:20.758 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:26:20.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-09T16:26:20.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-09T16:26:20.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-03-09T16:26:20.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:20.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:26:20.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-03-09T16:26:20.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-03-09T16:26:20.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173' 2024-03-09T16:26:20.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:20.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:26:21.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-09T16:26:21.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-09T16:26:21.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173 2-64424509462' 2024-03-09T16:26:21.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:21.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-03-09T16:26:21.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:21.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:26:21.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-03-09T16:26:21.071 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:21.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-09T16:26:21.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-03-09T16:26:21.073 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808834 2024-03-09T16:26:21.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103173 2024-03-09T16:26:21.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:26:21.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103173 -lt 249108103173 2024-03-09T16:26:21.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:21.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-09T16:26:21.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:21.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:26:21.640 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-09T16:26:21.640 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:21.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-09T16:26:21.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-09T16:26:21.642 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509462 2024-03-09T16:26:21.642 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:26:21.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-09T16:26:21.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:26:21.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:21.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:26:22.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:22.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:22.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:26:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:26:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:26:22.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:26:22.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:26:22.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:26:22.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:26:22.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:26:22.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:26:23.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:26:23.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:26:25.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:26:25.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:26:25.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:26:25.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:26:25.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:26:25.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:26:25.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:26:25.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:26:25.453 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:25.450+0000 7f92505bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:25.454 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:25.452+0000 7f92505bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:25.455 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:25.453+0000 7f92505bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:25.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:25.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:25.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:25.977+0000 7f92505bb640 -1 Falling back to public interface 2024-03-09T16:26:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:26:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:27.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:28.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:28.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:28.400+0000 7f92505bb640 -1 osd.2 69 log_to_monitors true 2024-03-09T16:26:28.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:29.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:29.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:29.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:26:29.639 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:26:29.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:29.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:29.980 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/789673867,v1:127.0.0.1:6819/789673867] [v2:127.0.0.1:6820/789673867,v1:127.0.0.1:6821/789673867] exists,up 88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:26:29.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:26:30.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:26:30.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:26:30.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:26:30.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:26:30.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:26:30.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:30.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:26:30.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-03-09T16:26:30.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-03-09T16:26:30.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837' 2024-03-09T16:26:30.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:30.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:26:30.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103176 2024-03-09T16:26:30.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103176 2024-03-09T16:26:30.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176' 2024-03-09T16:26:30.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:30.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:26:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-09T16:26:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-09T16:26:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176 2-317827579906' 2024-03-09T16:26:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:30.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808837 2024-03-09T16:26:30.900 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:30.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:26:30.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808837 2024-03-09T16:26:30.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:30.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-03-09T16:26:30.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-03-09T16:26:30.902 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808837 2024-03-09T16:26:30.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:31.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808836 -lt 287762808837 2024-03-09T16:26:31.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:26:32.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:26:32.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:32.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-03-09T16:26:32.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:32.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103176 2024-03-09T16:26:32.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:32.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:26:32.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103176 2024-03-09T16:26:32.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:32.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103176 2024-03-09T16:26:32.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103176' 2024-03-09T16:26:32.486 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103176 2024-03-09T16:26:32.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:26:32.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103176 -lt 249108103176 2024-03-09T16:26:32.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:32.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-09T16:26:32.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:32.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:26:32.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-09T16:26:32.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:32.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-09T16:26:32.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-09T16:26:32.760 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 317827579906 2024-03-09T16:26:32.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:26:33.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-09T16:26:33.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:26:33.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:33.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:33.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:26:33.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:33.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:33.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:26:33.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:33.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:33.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:34.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:26:34.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:26:34.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:26:34.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:26:34.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:26:34.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:26:36.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:26:36.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:26:36.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:26:36.491 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:26:36.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:26:36.519 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:36.518+0000 7f2b69416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:36.521 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:36.520+0000 7f2b69416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:36.522 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:36.521+0000 7f2b69416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:26:36.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:36.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:37.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:37.042 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:37.041+0000 7f2b69416640 -1 Falling back to public interface 2024-03-09T16:26:38.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:26:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:38.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:38.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:39.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:39.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:39.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:26:39.324 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:26:39.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:39.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:39.409 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:39.408+0000 7f2b69416640 -1 osd.2 77 log_to_monitors true 2024-03-09T16:26:39.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:26:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/1996524704,v1:127.0.0.1:6819/1996524704] [v2:127.0.0.1:6820/1996524704,v1:127.0.0.1:6821/1996524704] exists,up 88e8cf70-5022-43db-8499-6d7479e03437 2024-03-09T16:26:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:26:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:26:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:26:40.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:26:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:26:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:26:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:26:40.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:26:40.902 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:26:40.903 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:26:40.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:26:40.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:26:40.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:26:40.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:26:40.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:26:40.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:26:40.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:26:40.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:26:40.989 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:26:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:26:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:26:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:26:41.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:41.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:26:41.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808840 2024-03-09T16:26:41.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808840 2024-03-09T16:26:41.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840' 2024-03-09T16:26:41.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:41.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:26:41.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103180 2024-03-09T16:26:41.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103180 2024-03-09T16:26:41.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180' 2024-03-09T16:26:41.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:41.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:26:41.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-09T16:26:41.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-09T16:26:41.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103180 2-352187318274' 2024-03-09T16:26:41.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:41.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808840 2024-03-09T16:26:41.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:41.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:26:41.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808840 2024-03-09T16:26:41.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:41.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808840 2024-03-09T16:26:41.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808840' 2024-03-09T16:26:41.776 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808840 2024-03-09T16:26:41.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:42.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808839 -lt 287762808840 2024-03-09T16:26:42.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:26:43.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:26:43.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:43.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808839 -lt 287762808840 2024-03-09T16:26:43.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:26:44.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:26:44.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:44.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808840 -lt 287762808840 2024-03-09T16:26:44.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:44.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103180 2024-03-09T16:26:44.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:44.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:26:44.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103180 2024-03-09T16:26:44.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:44.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103180 2024-03-09T16:26:44.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103180' 2024-03-09T16:26:44.614 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103180 2024-03-09T16:26:44.615 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:26:44.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103180 -lt 249108103180 2024-03-09T16:26:44.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:44.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318274 2024-03-09T16:26:44.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:44.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:26:44.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318274 2024-03-09T16:26:44.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:44.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-09T16:26:44.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-03-09T16:26:44.890 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318274 2024-03-09T16:26:44.890 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:26:45.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-09T16:26:45.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:26:45.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:45.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:45.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:26:45.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:45.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:45.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:26:45.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:45.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:45.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:46.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:26:46.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:26:46.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:26:46.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T16:26:46.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:26:46.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-09T16:26:46.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-09T16:26:46.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:26:46.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:26:46.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-09T16:26:46.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:26:46.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:26:46.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:26:46.608 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:26:46.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:26:46.984 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:26:46.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:26:46.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:26:46.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:26:47.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-09T16:26:47.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:26:47.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:26:47.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:2+0 records in 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:2+0 records out 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.8558e-05 s, 59.7 MB/s 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:26:47.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:26:47.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:26:47.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:26:47.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:26:47.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:26:49.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:26:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:26:49.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:26:49.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:26:49.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:49.683+0000 7f9d25e04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:49.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:49.686+0000 7f9d25e04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:49.688 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:49.687+0000 7f9d25e04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:49.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:26:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:26:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:50.202+0000 7f9d25e04640 -1 Falling back to public interface 2024-03-09T16:26:50.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:51.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:51.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:51.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:26:51.211 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:51.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:51.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:51.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:52.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:52.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:52.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:26:52.495 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:26:52.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:52.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:52.672 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:26:52.671+0000 7f9d25e04640 -1 osd.1 87 log_to_monitors true 2024-03-09T16:26:52.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:26:53.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:26:54.076 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 88 last_clean_interval [58,87) [v2:127.0.0.1:6810/1080218598,v1:127.0.0.1:6811/1080218598] [v2:127.0.0.1:6812/1080218598,v1:127.0.0.1:6813/1080218598] exists,up ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:26:54.077 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:26:54.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:26:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:26:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:26:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:26:54.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:26:54.164 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:26:54.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:26:54.446 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:26:54.447 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:26:54.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:26:54.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:54.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:26:54.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808844 2024-03-09T16:26:54.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808844 2024-03-09T16:26:54.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844' 2024-03-09T16:26:54.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:54.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:26:54.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-09T16:26:54.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-09T16:26:54.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-390842023938' 2024-03-09T16:26:54.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:26:54.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:26:54.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318278 2024-03-09T16:26:54.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318278 2024-03-09T16:26:54.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-390842023938 2-352187318278' 2024-03-09T16:26:54.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:54.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808844 2024-03-09T16:26:54.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:54.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:26:54.779 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808844 2024-03-09T16:26:54.779 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:54.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808844 2024-03-09T16:26:54.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-03-09T16:26:54.781 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808844 2024-03-09T16:26:54.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:55.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808842 -lt 287762808844 2024-03-09T16:26:55.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:26:56.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:26:56.052 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:26:56.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808844 -lt 287762808844 2024-03-09T16:26:56.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:56.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023938 2024-03-09T16:26:56.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:56.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:26:56.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023938 2024-03-09T16:26:56.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:56.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-09T16:26:56.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-03-09T16:26:56.386 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 390842023938 2024-03-09T16:26:56.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:26:56.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023938 2024-03-09T16:26:56.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:26:56.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318278 2024-03-09T16:26:56.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:26:56.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:26:56.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318278 2024-03-09T16:26:56.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:26:56.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318278 2024-03-09T16:26:56.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318278' 2024-03-09T16:26:56.659 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318278 2024-03-09T16:26:56.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:26:56.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318278 2024-03-09T16:26:56.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:26:56.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:56.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:57.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:57.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:57.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:26:57.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:57.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:57.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:57.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:26:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:26:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:26:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:26:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:26:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:26:57.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:26:57.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:57.982 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:26:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:58.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:58.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:58.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:26:58.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:26:58.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:26:58.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:26:58.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:26:58.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:26:58.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:26:58.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:26:58.617 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:26:58.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:26:58.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:26:58.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:26:58.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:26:58.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:26:59.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:26:59.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:26:59.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:26:59.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:26:59.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:26:59.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:26:59.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:26:59.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:27:00.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:27:00.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:27:00.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:27:00.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:27:00.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:27:00.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:27:00.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:00.517 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:00.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:27:00.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:00.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:00.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:27:01.124 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:27:01.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:27:01.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:27:01.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-09T16:27:01.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:27:01.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-09T16:27:02.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:27:02.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:02.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:02.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:02.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:02.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:02.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:02.267 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:02.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:27:02.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:02.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:02.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:02.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:27:02.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:27:02.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:27:02.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:27:02.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:27:03.270 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:27:03.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:27:03.763 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:27:03.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:27:03.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:27:03.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:27:03.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:27:03.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:27:03.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:27:03.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:27:03.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:27:03.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:27:03.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:27:04.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:27:04.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:04.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:27:06.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:27:06.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:27:06.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:27:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:27:06.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:27:06.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:27:06.507 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:27:06.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:27:06.534 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:06.532+0000 7ffa90615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:06.536 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:06.535+0000 7ffa90615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:06.536+0000 7ffa90615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:06.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:27:06.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:27:06.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:27:06.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:07.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:07.062 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:07.061+0000 7ffa90615640 -1 Falling back to public interface 2024-03-09T16:27:08.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:08.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:08.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:27:08.056 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:08.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:08.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:08.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:09.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:27:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:27:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:09.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:09.472 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:09.471+0000 7ffa90615640 -1 osd.0 95 log_to_monitors true 2024-03-09T16:27:09.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:27:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:27:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:10.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:11.094 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 100 up_thru 91 down_at 96 last_clean_interval [67,95) [v2:127.0.0.1:6802/1758138601,v1:127.0.0.1:6803/1758138601] [v2:127.0.0.1:6804/1758138601,v1:127.0.0.1:6805/1758138601] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:27:11.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:27:11.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:27:11.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:27:11.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:27:11.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:27:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:27:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:27:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:27:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:27:11.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:27:11.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:11.616 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:27:11.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-09T16:27:11.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-09T16:27:11.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602' 2024-03-09T16:27:11.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:11.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:27:11.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023942 2024-03-09T16:27:11.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023942 2024-03-09T16:27:11.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023942' 2024-03-09T16:27:11.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:11.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:27:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318282 2024-03-09T16:27:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318282 2024-03-09T16:27:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023942 2-352187318282' 2024-03-09T16:27:12.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:12.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729602 2024-03-09T16:27:12.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:12.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:27:12.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729602 2024-03-09T16:27:12.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:12.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-09T16:27:12.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-03-09T16:27:12.032 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 429496729602 2024-03-09T16:27:12.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:12.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 429496729602 2024-03-09T16:27:12.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:27:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:27:13.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:13.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729602 -lt 429496729602 2024-03-09T16:27:13.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:13.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023942 2024-03-09T16:27:13.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:13.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:27:13.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023942 2024-03-09T16:27:13.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023942 2024-03-09T16:27:13.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023942' 2024-03-09T16:27:13.602 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 390842023942 2024-03-09T16:27:13.602 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:27:13.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023943 -lt 390842023942 2024-03-09T16:27:13.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:13.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318282 2024-03-09T16:27:13.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:13.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:27:13.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318282 2024-03-09T16:27:13.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:13.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318282 2024-03-09T16:27:13.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2024-03-09T16:27:13.888 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318282 2024-03-09T16:27:13.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:27:14.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318282 -lt 352187318282 2024-03-09T16:27:14.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:27:14.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:14.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:14.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:27:14.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:14.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:14.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:14.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:14.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:14.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:14.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:14.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:27:14.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:14.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:14.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:15.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:27:15.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:27:15.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:27:15.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:27:15.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:27:15.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:27:15.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:27:15.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:27:15.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:27:15.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:27:15.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:27:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:27:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:27:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:27:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:27:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:27:17.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:27:17.461 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:27:17.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:27:17.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:27:17.463 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:27:17.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:27:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:17.489+0000 7f1e00949640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:17.492 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:17.492+0000 7f1e00949640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:17.493+0000 7f1e00949640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:17.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:17.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:27:18.009 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:18.008+0000 7f1e00949640 -1 Falling back to public interface 2024-03-09T16:27:18.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:19.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:27:19.309 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:20.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:27:20.402 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:20.401+0000 7f1e00949640 -1 osd.1 103 log_to_monitors true 2024-03-09T16:27:20.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:21.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:27:22.155 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [91,103) [v2:127.0.0.1:6810/471922892,v1:127.0.0.1:6811/471922892] [v2:127.0.0.1:6812/471922892,v1:127.0.0.1:6813/471922892] exists,up ac32d04f-e700-4416-bcb1-3a5636eea15d 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:27:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:27:22.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:27:22.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:27:22.545 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:22.545 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:27:22.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:27:22.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:22.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:27:22.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729605 2024-03-09T16:27:22.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729605 2024-03-09T16:27:22.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605' 2024-03-09T16:27:22.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:22.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:27:22.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-03-09T16:27:22.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-03-09T16:27:22.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-463856467970' 2024-03-09T16:27:22.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:22.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:27:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318285 2024-03-09T16:27:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318285 2024-03-09T16:27:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-463856467970 2-352187318285' 2024-03-09T16:27:22.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:22.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729605 2024-03-09T16:27:22.899 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:22.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:27:22.900 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729605 2024-03-09T16:27:22.900 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:22.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729605 2024-03-09T16:27:22.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-03-09T16:27:22.901 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 429496729605 2024-03-09T16:27:22.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:23.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729605 2024-03-09T16:27:23.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:27:24.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:27:24.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:24.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729605 -lt 429496729605 2024-03-09T16:27:24.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:24.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-03-09T16:27:24.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:24.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:27:24.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-03-09T16:27:24.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:24.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-03-09T16:27:24.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-03-09T16:27:24.459 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 463856467970 2024-03-09T16:27:24.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:27:24.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-03-09T16:27:24.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:24.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318285 2024-03-09T16:27:24.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:24.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:27:24.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318285 2024-03-09T16:27:24.733 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:24.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318285 2024-03-09T16:27:24.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-03-09T16:27:24.734 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318285 2024-03-09T16:27:24.734 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:27:25.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318285 -lt 352187318285 2024-03-09T16:27:25.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:27:25.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:25.007 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:25.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:27:25.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:25.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:25.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:27:25.615 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:25.615 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:25.615 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:25.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:27:25.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:27:26.411 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:27:26.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:27:26.787 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:27:26.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:27:26.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:27:26.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:27:27.016 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-09T16:27:27.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:27:27.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:27:27.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T16:27:27.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T16:27:27.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:27:27.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:27:27.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:27.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:27:29.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:27:29.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:27:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:27:29.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:27:29.422 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:29.420+0000 7f2a426a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:29.424 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:29.422+0000 7f2a426a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:29.425 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:29.423+0000 7f2a426a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:29.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:29.944 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:29.942+0000 7f2a426a4640 -1 Falling back to public interface 2024-03-09T16:27:29.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:30.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:30.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:30.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:27:30.950 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:30.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:30.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:31.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:32.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:32.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:32.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:27:32.224 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:27:32.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:32.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:32.305 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:32.303+0000 7f2a426a4640 -1 osd.0 113 log_to_monitors true 2024-03-09T16:27:32.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:33.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:33.784 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 118 up_thru 118 down_at 114 last_clean_interval [100,113) [v2:127.0.0.1:6802/2586360407,v1:127.0.0.1:6803/2586360407] [v2:127.0.0.1:6804/2586360407,v1:127.0.0.1:6805/2586360407] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:27:33.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:27:33.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:27:33.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:27:33.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:27:33.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:27:33.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:27:33.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:27:33.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:27:33.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:27:34.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:27:34.216 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:34.217 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:27:34.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:27:34.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:34.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:27:34.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-09T16:27:34.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-09T16:27:34.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930' 2024-03-09T16:27:34.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:34.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:27:34.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467973 2024-03-09T16:27:34.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467973 2024-03-09T16:27:34.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930 1-463856467973' 2024-03-09T16:27:34.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:34.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:27:34.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318288 2024-03-09T16:27:34.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318288 2024-03-09T16:27:34.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930 1-463856467973 2-352187318288' 2024-03-09T16:27:34.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:34.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-506806140930 2024-03-09T16:27:34.527 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:34.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:27:34.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-506806140930 2024-03-09T16:27:34.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:34.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-03-09T16:27:34.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 506806140930' 2024-03-09T16:27:34.530 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 506806140930 2024-03-09T16:27:34.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:34.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 506806140930 2024-03-09T16:27:34.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:27:35.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:27:35.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:36.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140930 -lt 506806140930 2024-03-09T16:27:36.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:36.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467973 2024-03-09T16:27:36.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:36.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:27:36.083 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467973 2024-03-09T16:27:36.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467973 2024-03-09T16:27:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467973' 2024-03-09T16:27:36.085 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 463856467973 2024-03-09T16:27:36.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:27:36.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467973 -lt 463856467973 2024-03-09T16:27:36.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:36.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318288 2024-03-09T16:27:36.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:36.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:27:36.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318288 2024-03-09T16:27:36.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:36.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318288 2024-03-09T16:27:36.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318288' 2024-03-09T16:27:36.375 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318288 2024-03-09T16:27:36.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:27:36.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318289 -lt 352187318288 2024-03-09T16:27:36.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:27:36.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:36.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:37.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:27:37.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:37.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:37.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:27:37.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:27:37.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:37.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:27:40.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:27:40.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:27:40.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:27:40.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:27:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:27:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:27:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:27:40.335 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:27:40.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:27:40.362 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:40.361+0000 7fa025cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:40.364 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:40.363+0000 7fa025cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:40.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:40.364+0000 7fa025cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:27:40.621 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:27:40.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:40.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:40.885 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:40.884+0000 7fa025cd9640 -1 Falling back to public interface 2024-03-09T16:27:40.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:42.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:43.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:43.185 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:27:43.184+0000 7fa025cd9640 -1 osd.0 120 log_to_monitors true 2024-03-09T16:27:43.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:27:44.525 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 125 up_thru 125 down_at 121 last_clean_interval [118,120) [v2:127.0.0.1:6802/571246659,v1:127.0.0.1:6803/571246659] [v2:127.0.0.1:6804/571246659,v1:127.0.0.1:6805/571246659] exists,up e7b665b6-d518-4347-b205-96d56d660493 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:27:44.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:27:44.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:27:44.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:45.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:27:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=536870912002 2024-03-09T16:27:45.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 536870912002 2024-03-09T16:27:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002' 2024-03-09T16:27:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:45.339 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:27:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467976 2024-03-09T16:27:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467976 2024-03-09T16:27:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002 1-463856467976' 2024-03-09T16:27:45.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:27:45.442 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:27:45.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318292 2024-03-09T16:27:45.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318292 2024-03-09T16:27:45.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002 1-463856467976 2-352187318292' 2024-03-09T16:27:45.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:45.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-536870912002 2024-03-09T16:27:45.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:45.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:27:45.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-536870912002 2024-03-09T16:27:45.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:45.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=536870912002 2024-03-09T16:27:45.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 536870912002' 2024-03-09T16:27:45.550 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 536870912002 2024-03-09T16:27:45.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:45.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 536870912002 2024-03-09T16:27:45.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:27:46.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:27:46.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:47.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 536870912002 2024-03-09T16:27:47.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:27:48.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:27:48.096 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:27:48.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 536870912002 -lt 536870912002 2024-03-09T16:27:48.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:48.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467976 2024-03-09T16:27:48.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:48.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:27:48.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467976 2024-03-09T16:27:48.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:48.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467976 2024-03-09T16:27:48.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467976' 2024-03-09T16:27:48.384 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 463856467976 2024-03-09T16:27:48.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:27:48.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467976 2024-03-09T16:27:48.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:27:48.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318292 2024-03-09T16:27:48.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:27:48.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:27:48.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318292 2024-03-09T16:27:48.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:27:48.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318292 2024-03-09T16:27:48.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318292' 2024-03-09T16:27:48.659 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 352187318292 2024-03-09T16:27:48.660 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:27:48.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318292 -lt 352187318292 2024-03-09T16:27:48.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:27:48.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:48.932 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:49.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:27:49.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:49.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:49.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:49.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:49.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:49.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:49.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:49.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:27:49.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:49.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:49.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:49.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:27:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:27:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:27:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:27:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:27:49.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:27:49.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:27:50.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:50.000 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:50.000 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:50.000 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:50.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:50.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:50.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:50.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:27:50.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:50.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:50.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:27:50.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:27:50.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:27:50.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:27:50.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:27:50.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:27:50.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:27:51.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:27:51.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:27:51.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:27:51.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:27:51.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:27:51.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:27:51.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:27:51.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:27:51.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:27:51.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:27:51.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-09T16:27:51.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-09T16:27:51.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:27:51.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-09T16:27:51.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-09T16:27:51.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:27:52.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-09T16:27:52.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-09T16:27:52.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T16:27:52.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T16:27:52.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:52.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:52.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:52.033 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:52.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:52.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T16:27:52.555 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-09T16:27:52.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:52.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:27:52.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:52.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:52.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:52.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:52.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:27:53.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:27:53.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:53.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:27:53.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:53.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:53.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:53.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:54.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:54.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:27:55.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:27:55.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:55.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:27:55.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:55.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:55.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:55.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:55.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:55.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:27:56.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:27:56.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:56.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:27:56.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:56.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:56.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:56.396 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:56.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:56.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:27:57.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:27:57.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:57.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:27:57.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:57.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:27:57.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:57.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:27:57.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:27:52.567076+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:57.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:27:57.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T16:27:57.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:27:57.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-09T16:27:57.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:27:57.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T16:27:57.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T16:27:58.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:27:58.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=125 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:27:58.015 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:27:58.016 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:27:58.016 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:27:58.016 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:27:58.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:27:58.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:27:58.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:27:58.042 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:27:58.042 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:27:58.042 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:27:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:27:58.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:27:58.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:58.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:27:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:58.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T16:27:58.586 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-09T16:27:58.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T16:25:27.354493+0000 last_deep_scrub_stamp 2024-03-09T16:27:58.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:27:58.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:58.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:58.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:27:58.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:27:58.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:27:59.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:27:59.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:27:59.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:27:59.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:27:59.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:27:59.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:27:59.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:28:00.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:25:27.354493+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:28:00.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:28:01.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:28:01.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:28:01.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:28:01.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:28:01.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:28:01.149 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:28:01.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:28:01.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:27:58.793519+0000 '>' 2024-03-09T16:25:27.354493+0000 2024-03-09T16:28:01.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:28:01.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T16:28:01.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:28:01.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-09T16:28:01.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:28:01.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T16:28:01.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T16:28:01.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=125 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:28:01.491 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:28:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:28:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:28:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:28:01.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:28:01.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:28:01.515 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:28:01.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:28:01.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:28:01.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:28:01.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-09T16:28:01.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-09T16:28:01.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-09T16:28:01.906 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' removed 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:28:01.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:28:01.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:28:02.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:28:02.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:28:02.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:28:02.028 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:28:02.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:28:02.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:28:02.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:28:02.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:28:02.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:28:02.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:28:02.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:28:02.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:28:02.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:28:02.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:28:02.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:28:02.059 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.060 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:28:02.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:28:02.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:28:02.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:28:02.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:28:02.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:28:02.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:28:02.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:28:02.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:28:02.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:28:02.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:28:02.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:28:02.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:28:02.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:28:02.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:28:02.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:28:02.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:28:02.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:28:02.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:28:02.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:28:02.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:28:02.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:28:02.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:28:02.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:28:02.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:28:02.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:28:02.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:28:02.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-03-09T16:28:02.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:28:02.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:28:02.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:02.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:28:02.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:28:02.665 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:28:02.666 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:28:02.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:28:02.748 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:02.749 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:02.749 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:28:02.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:28:02.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:28:02.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-09T16:28:02.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:28:02.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:28:02.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:28:02.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:28:02.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:28:02.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:03.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:28:03.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:28:03.670 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:28:03.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:28:03.675 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:28:03.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 30197b28-6ba1-4853-a4d5-366599c9a1cc' 2024-03-09T16:28:03.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:28:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATjuxlwv0vKRAAesKRxSHUAKtUceC2ifKmZw== 2024-03-09T16:28:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATjuxlwv0vKRAAesKRxSHUAKtUceC2ifKmZw=="}' 2024-03-09T16:28:03.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30197b28-6ba1-4853-a4d5-366599c9a1cc -i td/osd-scrub-repair/0/new.json 2024-03-09T16:28:03.896 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:28:03.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:28:03.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATjuxlwv0vKRAAesKRxSHUAKtUceC2ifKmZw== --osd-uuid 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:28:03.935 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.934+0000 7fbf99a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:03.947 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.946+0000 7fbf99a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:03.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.947+0000 7fbf99a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:03.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.948+0000 7fbf99a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:28:03.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.948+0000 7fbf99a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:28:03.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:03.948+0000 7fbf99a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:28:04.039 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:04.038+0000 7fbf99a24640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:28:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:28:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:28:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:28:09.748 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:28:09.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:28:10.098 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:28:10.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:28:10.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:28:10.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:28:10.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:28:10.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:28:10.125 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:10.123+0000 7f2142e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:10.126 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:10.125+0000 7f2142e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:10.127 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:10.126+0000 7f2142e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:28:10.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:10.372 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:28:10.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:28:10.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:10.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:10.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:10.654 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:10.653+0000 7f2142e12640 -1 Falling back to public interface 2024-03-09T16:28:11.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:11.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:11.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:28:11.643 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:28:11.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:11.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:11.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:12.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:12.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:12.918 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:28:12.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:28:12.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:12.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:12.968 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:12.967+0000 7f2142e12640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:28:13.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:14.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:14.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:14.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:28:14.228 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:28:14.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:14.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:14.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:15.515 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:28:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:28:15.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:15.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:15.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:16.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:16.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:16.907 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:28:16.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:28:16.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:16.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:17.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:18.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:18.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:18.191 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:28:18.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:28:18.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:18.192 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:18.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:19.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:19.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:19.519 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:28:19.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:28:19.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:19.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:19.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:20.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:20.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:20.883 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:28:20.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:28:20.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:20.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:21.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:22.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:22.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:22.164 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:28:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:28:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:22.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:23.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:23.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:23.447 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:28:23.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:28:23.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:23.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:23.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:24.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:24.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:24.728 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:28:24.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:28:24.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:24.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:25.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:26.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:26.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:26.253 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:28:26.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:28:26.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:26.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:26.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:27.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:27.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:27.786 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:28:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:28:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:27.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:28.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:29.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:29.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:29.698 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:28:29.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:28:29.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:29.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:30.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:31.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:31.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:31.199 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:28:31.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:28:31.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:31.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:31.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:32.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:32.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:32.595 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:28:32.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:28:32.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:32.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:33.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:34.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:34.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:34.299 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:28:34.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:28:34.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:34.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:34.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:35.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:35.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:35.656 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:28:35.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:28:35.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:35.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:28:35.941 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/381537872,v1:127.0.0.1:6803/381537872] [v2:127.0.0.1:6804/381537872,v1:127.0.0.1:6805/381537872] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:28:35.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:28:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:28:35.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:28:35.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:28:35.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:28:35.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21' 2024-03-09T16:28:35.949 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:28:35.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:28:35.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzjuxl3qxUORAAbJDGy2ecCtgdRCp9jmMUzw== 2024-03-09T16:28:35.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzjuxl3qxUORAAbJDGy2ecCtgdRCp9jmMUzw=="}' 2024-03-09T16:28:35.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:28:36.323 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:28:36.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:28:36.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzjuxl3qxUORAAbJDGy2ecCtgdRCp9jmMUzw== --osd-uuid 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:28:36.414 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.361+0000 7f70c0ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:36.525 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.522+0000 7f70c0ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:36.525 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.524+0000 7f70c0ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:36.526 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.524+0000 7f70c0ca8640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:28:36.526 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.524+0000 7f70c0ca8640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:28:36.526 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.524+0000 7f70c0ca8640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:28:36.624 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:36.623+0000 7f70c0ca8640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:28:42.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:28:42.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:28:42.446 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:28:42.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:28:42.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:28:42.800 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:28:42.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:28:42.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:28:42.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:28:42.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:28:42.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:28:42.825 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:42.823+0000 7f163f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:42.827 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:42.825+0000 7f163f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:42.828 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:42.826+0000 7f163f811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:28:43.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:43.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:28:43.074 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:28:43.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:43.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:43.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:43.355 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:43.353+0000 7f163f811640 -1 Falling back to public interface 2024-03-09T16:28:44.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:44.340 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:28:44.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:44.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:28:44.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:44.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:44.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:45.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:45.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:45.621 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:28:45.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:28:45.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:45.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:45.686 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:28:45.684+0000 7f163f811640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:28:45.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:46.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:46.947 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:28:46.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:46.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:28:46.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:46.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:47.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:48.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:48.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:48.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:28:48.267 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:28:48.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:48.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:48.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:49.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:49.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:49.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:28:49.979 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:28:49.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:49.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:49.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:50.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:50.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:50.890 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:28:50.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:28:50.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:50.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:51.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:52.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:52.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:52.173 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:28:52.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:28:52.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:52.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:52.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:53.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:53.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:53.463 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:28:53.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:28:53.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:53.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:53.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:54.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:54.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:54.817 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:28:54.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:28:54.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:54.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:55.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:56.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:56.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:56.102 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:28:56.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:28:56.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:56.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:56.528 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:57.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:57.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:57.529 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:28:57.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:28:57.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:57.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:57.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:28:58.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:28:58.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:28:58.813 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:28:58.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:28:58.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:28:58.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:28:59.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:00.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:00.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:00.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:29:00.413 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:29:00.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:00.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:00.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:01.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:01.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:01.735 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:29:01.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:29:01.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:01.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:02.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:03.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:03.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:03.120 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:29:03.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:29:03.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:03.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:03.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:04.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:04.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:04.551 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:29:04.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:29:04.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:04.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:05.013 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:06.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:06.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:06.015 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:29:06.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:29:06.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:06.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:06.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:07.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:07.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:07.642 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:29:07.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:29:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:08.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:09.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:09.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:09.049 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:29:09.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:29:09.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:09.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:29:09.331 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2437566844,v1:127.0.0.1:6811/2437566844] [v2:127.0.0.1:6812/2437566844,v1:127.0.0.1:6813/2437566844] exists,up 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:29:09.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:29:09.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:29:09.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 452d1a58-190f-48bf-a0ee-b2c9c01389f6' 2024-03-09T16:29:09.338 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:29:09.339 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:29:09.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVjuxlfATrFBAAmprs8qLUKvBujuaXMIkXTg== 2024-03-09T16:29:09.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVjuxlfATrFBAAmprs8qLUKvBujuaXMIkXTg=="}' 2024-03-09T16:29:09.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 452d1a58-190f-48bf-a0ee-b2c9c01389f6 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:29:09.712 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:29:09.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:29:09.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVjuxlfATrFBAAmprs8qLUKvBujuaXMIkXTg== --osd-uuid 452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:29:09.752 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.750+0000 7f9611c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:09.796 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.795+0000 7f9611c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:09.797 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.796+0000 7f9611c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:09.797 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.796+0000 7f9611c48640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:29:09.798 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.796+0000 7f9611c48640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:29:09.798 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.796+0000 7f9611c48640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:29:09.871 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:09.870+0000 7f9611c48640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:29:16.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:29:16.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:29:16.038 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:29:16.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:29:16.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:29:16.390 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:29:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:29:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:29:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:29:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:29:16.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:29:16.416 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:16.415+0000 7f861b9fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:16.418 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:16.417+0000 7f861b9fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:16.420 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:16.419+0000 7f861b9fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:29:16.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:29:16.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:29:16.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:29:16.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:29:16.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:29:16.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:16.664 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:29:16.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:29:16.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:16.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:16.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:16.944 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:16.943+0000 7f861b9fa640 -1 Falling back to public interface 2024-03-09T16:29:17.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:17.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:17.937 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:29:17.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:29:17.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:17.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:18.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:19.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:19.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:19.218 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:29:19.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:29:19.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:19.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:19.302 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:29:19.301+0000 7f861b9fa640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:29:19.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:20.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:20.503 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:29:20.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:20.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:29:20.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:20.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:20.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:21.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:21.788 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:29:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:29:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:21.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:22.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:23.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:23.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:23.131 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:29:23.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:29:23.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:23.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:23.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:24.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:24.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:24.411 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:29:24.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:29:24.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:24.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:24.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:25.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:25.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:25.692 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:29:25.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:29:25.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:25.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:25.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:26.978 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:29:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:29:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:26.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:27.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:28.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:28.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:28.262 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:29:28.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:29:28.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:28.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:28.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:29.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:29.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:29.628 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:29:29.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:29:29.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:29.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:29.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:30.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:30.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:30.917 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:29:30.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:29:30.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:30.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:31.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:32.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:32.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:32.546 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:29:32.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:29:32.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:32.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:33.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:34.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:34.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:34.176 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:29:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:29:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:34.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:34.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:35.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:35.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:35.509 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:29:35.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:29:35.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:35.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:35.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:36.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:36.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:36.867 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:29:36.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:29:36.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:36.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:37.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:38.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:38.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:38.482 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:29:38.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:29:38.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:38.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:39.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:40.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:40.225 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:29:40.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:29:40.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:40.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:40.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:41.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:41.568 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:29:41.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:29:41.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:41.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:41.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:29:42.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:29:42.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:29:42.871 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:29:42.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:29:42.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:29:42.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:29:43.143 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1804911598,v1:127.0.0.1:6819/1804911598] [v2:127.0.0.1:6820/1804911598,v1:127.0.0.1:6821/1804911598] exists,up 452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:29:43.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:29:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:29:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:29:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-09T16:29:43.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:29:43.407 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:29:43.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:29:43.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:29:43.820 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:29:43.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:29:44.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:29:46.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-09T16:29:46.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T16:29:46.901 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'foo' created 2024-03-09T16:29:46.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:29:47.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-03-09T16:29:47.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:29:47.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:29:47.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T16:29:47.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:29:47.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-09T16:29:48.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:29:48.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:29:49.162 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:29:49.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:29:50.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T16:29:50.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T16:29:50.753 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 3 allow_ec_overwrites to true 2024-03-09T16:29:50.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:29:50.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:29:50.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:29:50.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:29:50.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:29:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:29:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:29:50.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:29:51.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:29:51.122 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:29:51.123 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:29:51.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:29:51.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:51.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:29:51.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-09T16:29:51.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-09T16:29:51.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-09T16:29:51.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:51.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:29:51.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T16:29:51.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T16:29:51.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-03-09T16:29:51.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:51.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:29:51.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T16:29:51.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T16:29:51.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509450' 2024-03-09T16:29:51.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:51.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-09T16:29:51.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:51.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:29:51.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-09T16:29:51.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:51.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-09T16:29:51.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-09T16:29:51.461 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836499 2024-03-09T16:29:51.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:29:51.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836499 2024-03-09T16:29:51.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:29:52.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:29:52.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:29:53.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-09T16:29:53.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:53.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T16:29:53.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:53.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:29:53.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T16:29:53.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:53.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T16:29:53.047 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672973 2024-03-09T16:29:53.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T16:29:53.048 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:29:53.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T16:29:53.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:53.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T16:29:53.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:53.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:29:53.319 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T16:29:53.319 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:53.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T16:29:53.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T16:29:53.320 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509450 2024-03-09T16:29:53.321 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:29:53.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T16:29:53.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:29:53.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:29:53.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:29:53.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:29:53.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:29:53.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:29:54.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:29:54.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:29:54.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:29:54.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:29:54.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:29:54.585 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:29:54.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:29:54.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:29:54.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:29:54.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:29:54.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:29:54.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:29:54.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:29:54.945 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:29:54.946 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:29:54.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:29:54.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:54.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:29:55.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T16:29:55.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T16:29:55.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T16:29:55.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:55.050 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:29:55.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T16:29:55.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T16:29:55.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-03-09T16:29:55.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:29:55.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509452' 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T16:29:55.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:55.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:29:55.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T16:29:55.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:55.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T16:29:55.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T16:29:55.259 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T16:29:55.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:29:55.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T16:29:55.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:29:56.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:29:56.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:29:56.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836500 2024-03-09T16:29:56.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:56.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T16:29:56.813 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:56.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:29:56.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T16:29:56.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:56.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T16:29:56.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T16:29:56.816 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672975 2024-03-09T16:29:56.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:29:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T16:29:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:29:57.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-09T16:29:57.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:29:57.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:29:57.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-09T16:29:57.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:29:57.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-09T16:29:57.090 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-09T16:29:57.091 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509452 2024-03-09T16:29:57.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:29:57.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-09T16:29:57.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:29:57.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:29:57.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:29:57.699 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:29:57.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:29:57.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:29:57.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:29:57.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:29:57.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:29:57.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:29:58.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:29:58.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:29:58.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:29:58.328 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:29:58.330 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:29:58.709 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:29:58.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:29:59.111 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:29:59.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:29:59.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:29:59.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:29:59.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-09T16:29:59.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:29:59.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:29:59.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-09T16:29:59.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-09T16:29:59.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:29:59.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:29:59.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:30:00.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:30:00.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:30:00.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:02.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:30:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:30:02.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:02.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:30:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:30:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:30:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:30:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:30:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:30:02.433 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:02.432+0000 7fa072823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:02.438 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:02.434+0000 7fa072823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:02.438 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:02.436+0000 7fa072823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:02.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:30:02.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:02.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:02.955 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:02.954+0000 7fa072823640 -1 Falling back to public interface 2024-03-09T16:30:02.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:03.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:04.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:05.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:05.329 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:05.328+0000 7fa072823640 -1 osd.1 35 log_to_monitors true 2024-03-09T16:30:05.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:06.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3201005775,v1:127.0.0.1:6811/3201005775] [v2:127.0.0.1:6812/3201005775,v1:127.0.0.1:6813/3201005775] exists,up 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:30:06.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:30:06.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:30:07.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:30:07.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:30:07.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:30:07.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:30:07.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:30:07.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:07.326 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:30:07.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-09T16:30:07.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-09T16:30:07.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-09T16:30:07.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:07.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:30:07.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T16:30:07.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T16:30:07.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-171798691842' 2024-03-09T16:30:07.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:07.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-171798691842 2-64424509455' 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-09T16:30:07.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:07.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:30:07.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-09T16:30:07.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:07.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-09T16:30:07.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-09T16:30:07.646 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836504 2024-03-09T16:30:07.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:08.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-09T16:30:08.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:08.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-09T16:30:08.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:08.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:30:08.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-09T16:30:08.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T16:30:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-09T16:30:08.097 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 171798691842 2024-03-09T16:30:08.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:30:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T16:30:08.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:08.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T16:30:08.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:08.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:30:08.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T16:30:08.371 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:08.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T16:30:08.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T16:30:08.372 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509455 2024-03-09T16:30:08.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:30:08.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T16:30:08.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:30:08.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:08.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:09.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:30:09.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:09.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:09.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:30:09.308 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:09.309 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:09.309 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:30:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:30:09.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:30:09.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:09.744 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:09.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:10.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:30:10.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:10.016 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:10.016 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:10.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:30:10.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:30:10.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:30:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:30:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:30:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:30:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:30:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:30:10.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:30:10.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:10.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:11.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:30:11.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:11.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:11.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:30:11.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:30:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:30:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:30:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:30:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:30:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:30:12.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:30:12.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:12.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:12.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:30:12.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:12.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:12.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T16:30:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T16:30:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:30:12.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:30:12.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:30:12.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:30:12.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:30:12.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:30:12.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:30:13.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:30:13.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:30:13.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-09T16:30:13.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:30:13.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-09T16:30:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:30:14.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:14.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:14.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:30:14.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:14.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:14.327 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:14.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:30:14.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:30:14.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:30:14.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:30:14.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-09T16:30:14.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-09T16:30:14.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:30:14.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:30:14.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-09T16:30:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:30:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:30:14.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:30:15.032 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:30:15.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:30:15.514 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:30:15.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:30:15.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:30:15.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:30:15.743 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-09T16:30:15.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:30:15.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:30:15.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-09T16:30:15.746 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.2223e-05 s, 32.9 MB/s 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:30:15.747 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:30:15.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:30:15.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:30:15.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:30:17.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:30:18.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:30:18.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:30:18.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:30:18.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:30:18.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:30:18.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:30:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:30:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:30:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:30:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:30:18.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:30:18.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:30:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:30:18.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:30:18.887 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:18.886+0000 7f40c9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:18.889 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:18.888+0000 7f40c9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:18.890 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:18.889+0000 7f40c9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:19.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:30:19.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:30:19.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:30:19.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:30:19.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:30:19.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:19.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:30:19.140 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:30:19.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:19.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:19.404 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:19.403+0000 7f40c9c14640 -1 Falling back to public interface 2024-03-09T16:30:19.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:20.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:20.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:20.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:30:20.412 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:20.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:20.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:20.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:21.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:30:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:30:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:21.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:21.824 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:21.823+0000 7f40c9c14640 -1 osd.0 44 log_to_monitors true 2024-03-09T16:30:22.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:23.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 48 up_thru 40 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/2928935648,v1:127.0.0.1:6803/2928935648] [v2:127.0.0.1:6804/2928935648,v1:127.0.0.1:6805/2928935648] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:30:23.437 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:30:23.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:30:23.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:30:23.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:30:23.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:30:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:30:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:30:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:30:23.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:30:23.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:23.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:30:24.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-09T16:30:24.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-09T16:30:24.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-03-09T16:30:24.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:24.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:30:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-03-09T16:30:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-03-09T16:30:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691846' 2024-03-09T16:30:24.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:24.207 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:30:24.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-09T16:30:24.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-09T16:30:24.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691846 2-64424509460' 2024-03-09T16:30:24.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:24.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-03-09T16:30:24.313 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:24.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:30:24.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-03-09T16:30:24.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:24.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-09T16:30:24.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-03-09T16:30:24.315 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 206158430210 2024-03-09T16:30:24.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:24.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-03-09T16:30:24.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:30:25.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:30:25.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:25.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-09T16:30:25.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:25.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-03-09T16:30:25.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:25.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:30:25.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-03-09T16:30:25.885 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:25.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-03-09T16:30:25.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-03-09T16:30:25.886 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 171798691846 2024-03-09T16:30:25.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:30:26.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691846 2024-03-09T16:30:26.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:26.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-09T16:30:26.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:26.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:30:26.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-09T16:30:26.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:26.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-09T16:30:26.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-09T16:30:26.161 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509460 2024-03-09T16:30:26.161 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:30:26.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-09T16:30:26.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:30:26.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:26.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:26.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:30:26.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:27.033 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:30:27.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:27.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:27.034 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-09T16:30:27.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:30:27.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:30:27.739 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:30:27.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:30:28.115 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:30:28.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:30:28.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:30:28.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:30:28.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-09T16:30:28.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:30:28.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:30:28.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:30:28.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:30:28.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T16:30:28.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:30:28.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:30:28.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:30:28.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:30:28.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:30:28.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-09T16:30:30.379 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:30:30.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:30.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:30:30.899 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:30:30.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:30:30.900 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:30:30.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:30:30.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:30:30.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:30:30.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:30:30.926 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:30.925+0000 7fe6b4ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:30.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:30.928+0000 7fe6b4ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:30.930 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:30.929+0000 7fe6b4ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:31.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:30:31.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:30:31.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:30:31.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:30:31.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:30:31.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:31.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:30:31.186 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:30:31.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:31.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:31.445 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:31.444+0000 7fe6b4ddf640 -1 Falling back to public interface 2024-03-09T16:30:31.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:32.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:32.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:32.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:30:32.457 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:32.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:32.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:32.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:33.720 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:33.719+0000 7fe6b4ddf640 -1 osd.1 53 log_to_monitors true 2024-03-09T16:30:33.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:30:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:30:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:33.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:34.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:35.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:30:35.421 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/3790480897,v1:127.0.0.1:6811/3790480897] [v2:127.0.0.1:6812/3790480897,v1:127.0.0.1:6813/3790480897] exists,up 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:30:35.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:30:35.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:30:35.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:30:35.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:30:35.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:30:35.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:30:35.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:30:35.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:30:35.423 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:30:35.423 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:30:35.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:30:35.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:30:35.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:30:35.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:30:35.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:30:35.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:30:35.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:30:35.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:30:35.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:30:35.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:30:35.782 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:35.782 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:30:35.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:30:35.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:35.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:30:35.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-03-09T16:30:35.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-03-09T16:30:35.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213' 2024-03-09T16:30:35.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:35.904 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:30:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-03-09T16:30:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-03-09T16:30:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-249108103170' 2024-03-09T16:30:36.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:36.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:30:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-09T16:30:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-09T16:30:36.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-249108103170 2-64424509463' 2024-03-09T16:30:36.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:36.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430213 2024-03-09T16:30:36.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:36.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:30:36.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430213 2024-03-09T16:30:36.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:36.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-03-09T16:30:36.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430213' 2024-03-09T16:30:36.114 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 206158430213 2024-03-09T16:30:36.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:36.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-03-09T16:30:36.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:30:37.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:30:37.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:37.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-03-09T16:30:37.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:30:38.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:30:38.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:38.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430213 2024-03-09T16:30:38.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:38.953 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-03-09T16:30:38.953 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:38.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:30:38.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-03-09T16:30:38.955 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:38.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-03-09T16:30:38.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-03-09T16:30:38.956 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103170 2024-03-09T16:30:38.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:30:39.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-03-09T16:30:39.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:39.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-09T16:30:39.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:39.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:30:39.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-09T16:30:39.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:39.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-09T16:30:39.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-09T16:30:39.233 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509463 2024-03-09T16:30:39.233 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:30:39.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-09T16:30:39.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:30:39.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:39.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:40.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:40.004 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:40.272 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:30:40.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:40.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:40.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:40.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:30:40.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:30:40.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-09T16:30:40.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:30:40.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:30:40.616 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:30:41.107 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:30:41.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:30:41.483 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:30:41.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:30:41.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:30:41.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:30:41.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-09T16:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:30:41.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-09T16:30:42.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:30:42.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:30:42.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:30:42.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:30:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:30:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:30:44.761 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:30:44.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:30:44.762 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:30:44.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:30:44.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:30:44.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:30:44.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:30:44.789 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:44.787+0000 7f3f74415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:44.791 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:44.789+0000 7f3f74415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:44.792 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:44.791+0000 7f3f74415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:30:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:45.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:45.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:45.315+0000 7f3f74415640 -1 Falling back to public interface 2024-03-09T16:30:46.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:46.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:46.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:30:46.316 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:46.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:46.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:46.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:47.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:47.615 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:47.614+0000 7f3f74415640 -1 osd.0 62 log_to_monitors true 2024-03-09T16:30:48.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:49.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:49.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:49.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:30:49.009 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:30:49.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:49.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [48,62) [v2:127.0.0.1:6802/1849010902,v1:127.0.0.1:6803/1849010902] [v2:127.0.0.1:6804/1849010902,v1:127.0.0.1:6805/1849010902] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:30:49.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:30:49.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:30:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:30:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:30:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:30:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:30:49.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:30:49.481 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:30:49.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:30:49.887 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:49.888 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:30:49.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:30:49.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:49.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:30:49.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-09T16:30:49.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-09T16:30:49.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-03-09T16:30:49.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:49.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:30:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103174 2024-03-09T16:30:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103174 2024-03-09T16:30:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103174' 2024-03-09T16:30:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:30:50.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:30:50.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-09T16:30:50.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-09T16:30:50.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103174 2-64424509467' 2024-03-09T16:30:50.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:50.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-03-09T16:30:50.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:30:50.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-03-09T16:30:50.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:50.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-09T16:30:50.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-03-09T16:30:50.200 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808834 2024-03-09T16:30:50.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:50.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 287762808834 2024-03-09T16:30:50.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:30:51.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:30:51.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:51.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 287762808834 2024-03-09T16:30:51.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:30:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:30:52.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:30:53.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-09T16:30:53.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:53.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103174 2024-03-09T16:30:53.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:30:53.054 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103174 2024-03-09T16:30:53.055 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103174 2024-03-09T16:30:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103174' 2024-03-09T16:30:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103174 2024-03-09T16:30:53.056 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:30:53.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103174 -lt 249108103174 2024-03-09T16:30:53.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:30:53.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-09T16:30:53.331 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:30:53.332 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:30:53.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-09T16:30:53.333 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:30:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-09T16:30:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-09T16:30:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 64424509467 2024-03-09T16:30:53.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:30:53.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-09T16:30:53.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:30:53.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:53.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:53.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:30:53.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:30:53.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:30:53.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:30:53.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:30:53.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:30:53.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:30:53.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:30:54.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:30:54.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:30:54.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:30:54.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:30:54.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:30:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:30:54.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:30:54.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:30:54.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:30:54.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:30:54.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:30:54.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:30:54.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:30:54.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T16:30:57.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:30:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:30:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:30:57.086 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:30:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:30:57.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:30:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:30:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:30:57.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:30:57.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:30:57.113 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:57.111+0000 7fee60614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:57.115 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:57.113+0000 7fee60614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:57.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:57.114+0000 7fee60614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:57.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:30:57.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:57.652 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:30:57.650+0000 7fee60614640 -1 Falling back to public interface 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:58.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:30:58.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:30:59.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:30:59.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:30:59.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:30:59.909 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:30:59.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:30:59.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:00.157 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:00.155+0000 7fee60614640 -1 osd.2 69 log_to_monitors true 2024-03-09T16:31:00.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:01.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:01.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:01.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:01.260 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:01.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:01.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:01.693 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1791160599,v1:127.0.0.1:6819/1791160599] [v2:127.0.0.1:6820/1791160599,v1:127.0.0.1:6821/1791160599] exists,up 452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:31:01.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:01.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:01.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:02.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:02.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-03-09T16:31:02.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-03-09T16:31:02.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837' 2024-03-09T16:31:02.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:02.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:02.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103177 2024-03-09T16:31:02.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103177 2024-03-09T16:31:02.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103177' 2024-03-09T16:31:02.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:02.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:02.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-09T16:31:02.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-09T16:31:02.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103177 2-317827579906' 2024-03-09T16:31:02.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:02.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808837 2024-03-09T16:31:02.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:02.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:02.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808837 2024-03-09T16:31:02.493 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:02.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-03-09T16:31:02.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-03-09T16:31:02.494 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808837 2024-03-09T16:31:02.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:02.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808836 -lt 287762808837 2024-03-09T16:31:02.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:31:03.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:31:03.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:04.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808838 -lt 287762808837 2024-03-09T16:31:04.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:04.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103177 2024-03-09T16:31:04.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:04.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:31:04.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103177 2024-03-09T16:31:04.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103177 2024-03-09T16:31:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-03-09T16:31:04.095 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103177 2024-03-09T16:31:04.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:31:04.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103177 -lt 249108103177 2024-03-09T16:31:04.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:04.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-09T16:31:04.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:04.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:31:04.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-09T16:31:04.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:04.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-09T16:31:04.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-09T16:31:04.369 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 317827579906 2024-03-09T16:31:04.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:31:04.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-09T16:31:04.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:31:04.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:04.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:04.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:31:04.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:31:05.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:31:05.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:31:05.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:05.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:05.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:31:05.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:31:05.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:31:05.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-09T16:31:05.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:31:05.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:31:05.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:31:05.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:31:05.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T16:31:08.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T16:31:08.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:31:08.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:31:08.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:31:08.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:31:08.045 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:08.042+0000 7fcd9da14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:08.047 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:08.045+0000 7fcd9da14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:08.048 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:08.046+0000 7fcd9da14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:08.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:31:08.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:31:08.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:08.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:08.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:08.576 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:08.574+0000 7fcd9da14640 -1 Falling back to public interface 2024-03-09T16:31:09.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:09.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:31:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:09.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:10.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:10.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:10.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:31:10.851 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:31:10.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:10.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:10.897 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:10.895+0000 7fcd9da14640 -1 osd.2 76 log_to_monitors true 2024-03-09T16:31:11.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:12.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:12.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:12.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:12.134 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:12.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:12.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:31:12.432 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/1728607525,v1:127.0.0.1:6819/1728607525] [v2:127.0.0.1:6820/1728607525,v1:127.0.0.1:6821/1728607525] exists,up 452d1a58-190f-48bf-a0ee-b2c9c01389f6 2024-03-09T16:31:12.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:12.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:12.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:12.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:12.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:12.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:12.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:12.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:12.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:12.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:12.521 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:12.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:12.807 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:12.807 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:12.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:12.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:12.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:12.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808841 2024-03-09T16:31:12.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808841 2024-03-09T16:31:12.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808841' 2024-03-09T16:31:12.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:12.912 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:13.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103180 2024-03-09T16:31:13.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103180 2024-03-09T16:31:13.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808841 1-249108103180' 2024-03-09T16:31:13.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:13.016 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808841 1-249108103180 2-347892350978' 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808841 2024-03-09T16:31:13.119 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:13.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:13.121 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808841 2024-03-09T16:31:13.121 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:13.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808841 2024-03-09T16:31:13.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-03-09T16:31:13.122 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808841 2024-03-09T16:31:13.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:13.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808839 -lt 287762808841 2024-03-09T16:31:13.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:31:14.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:31:14.391 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:14.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808841 -lt 287762808841 2024-03-09T16:31:14.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:14.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103180 2024-03-09T16:31:14.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:14.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:31:14.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103180 2024-03-09T16:31:14.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:14.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103180 2024-03-09T16:31:14.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103180' 2024-03-09T16:31:14.676 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 249108103180 2024-03-09T16:31:14.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:31:14.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103180 -lt 249108103180 2024-03-09T16:31:14.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:14.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-03-09T16:31:14.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:14.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:31:14.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-03-09T16:31:14.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:14.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-09T16:31:14.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-03-09T16:31:14.950 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350978 2024-03-09T16:31:14.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:31:15.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-03-09T16:31:15.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:31:15.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:15.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:15.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:31:15.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:31:15.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:31:15.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:31:15.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:31:15.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:31:15.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:31:15.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:31:15.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:31:15.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:31:15.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:15.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:31:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:31:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:31:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T16:31:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:31:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-09T16:31:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-09T16:31:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:31:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:31:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-09T16:31:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:31:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:31:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:31:16.574 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:31:16.585 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:31:17.076 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:31:17.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:31:17.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:31:17.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:31:17.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-09T16:31:17.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:31:17.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:31:17.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:2+0 records in 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:2+0 records out 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.9293e-05 s, 51.7 MB/s 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:31:17.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:31:17.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:31:17.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:31:17.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:31:19.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:31:19.683 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:31:19.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:31:19.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:31:19.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:31:19.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:31:19.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:31:19.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:31:19.687 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:31:19.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:31:19.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:31:19.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:31:19.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:31:19.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:31:19.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:31:19.713 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:19.712+0000 7f7e336d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:19.715 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:19.714+0000 7f7e336d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:19.717 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:19.716+0000 7f7e336d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:19.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:20.235 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:20.234+0000 7f7e336d1640 -1 Falling back to public interface 2024-03-09T16:31:20.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:21.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:21.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:21.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:31:21.242 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:21.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:21.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:21.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:22.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:22.569 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:22.568+0000 7f7e336d1640 -1 osd.1 85 log_to_monitors true 2024-03-09T16:31:22.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:23.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:23.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:23.886 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:23.886 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:23.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:23.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:24.166 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [58,85) [v2:127.0.0.1:6810/2256560668,v1:127.0.0.1:6811/2256560668] [v2:127.0.0.1:6812/2256560668,v1:127.0.0.1:6813/2256560668] exists,up 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:31:24.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:24.167 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:24.168 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:24.168 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:24.168 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:24.168 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:24.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:24.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:24.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:24.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:24.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:24.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:24.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:24.555 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:24.555 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:24.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:24.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:24.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:24.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808844 2024-03-09T16:31:24.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808844 2024-03-09T16:31:24.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844' 2024-03-09T16:31:24.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:24.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:24.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-09T16:31:24.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-09T16:31:24.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-386547056642' 2024-03-09T16:31:24.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:24.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350981 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350981 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808844 1-386547056642 2-347892350981' 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808844 2024-03-09T16:31:24.871 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:24.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:24.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808844 2024-03-09T16:31:24.872 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:24.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808844 2024-03-09T16:31:24.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-03-09T16:31:24.875 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 287762808844 2024-03-09T16:31:24.875 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808843 -lt 287762808844 2024-03-09T16:31:25.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:31:26.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:31:26.147 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:26.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808844 -lt 287762808844 2024-03-09T16:31:26.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:26.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-03-09T16:31:26.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:26.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:31:26.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-03-09T16:31:26.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:26.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-09T16:31:26.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-03-09T16:31:26.431 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 386547056642 2024-03-09T16:31:26.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:31:26.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-03-09T16:31:26.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:26.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350981 2024-03-09T16:31:26.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:26.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:31:26.702 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350981 2024-03-09T16:31:26.702 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:26.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350981 2024-03-09T16:31:26.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350981' 2024-03-09T16:31:26.703 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350981 2024-03-09T16:31:26.703 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:31:26.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350981 -lt 347892350981 2024-03-09T16:31:26.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:31:26.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:26.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:27.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:31:27.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:31:27.311 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:31:27.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:31:27.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:31:27.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:31:27.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:31:27.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:31:27.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:31:27.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:31:27.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:27.578 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:31:27.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:31:27.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:31:28.333 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:31:28.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:31:28.835 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:31:28.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:31:28.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:31:28.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:31:28.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-09T16:31:28.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:31:28.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:31:29.000 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:31:29.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:31:29.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:31:29.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:31:29.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-09T16:31:31.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:31:31.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:31:31.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:31:31.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:31:31.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:31:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:31:31.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:31:31.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:31:31.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:31:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:31:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:31:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:31:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:31:31.494 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:31.492+0000 7fd713c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:31.507 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:31.495+0000 7fd713c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:31.508 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:31.496+0000 7fd713c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:31.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:31:31.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:31:31.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:31.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:32.015 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:32.014+0000 7fd713c13640 -1 Falling back to public interface 2024-03-09T16:31:32.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:33.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:34.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:34.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:34.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:31:34.301 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:31:34.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:34.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:34.473 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:34.472+0000 7fd713c13640 -1 osd.0 94 log_to_monitors true 2024-03-09T16:31:34.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:35.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:35.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:35.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:35.590 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:35.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:35.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:36.014 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 99 up_thru 99 down_at 95 last_clean_interval [67,94) [v2:127.0.0.1:6802/3040583458,v1:127.0.0.1:6803/3040583458] [v2:127.0.0.1:6804/3040583458,v1:127.0.0.1:6805/3040583458] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:31:36.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:36.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:36.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:36.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:36.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:36.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:36.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:36.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:36.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:36.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:36.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-09T16:31:36.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-09T16:31:36.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306' 2024-03-09T16:31:36.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:36.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-03-09T16:31:36.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-03-09T16:31:36.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645' 2024-03-09T16:31:36.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:36.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:36.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350985 2024-03-09T16:31:36.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350985 2024-03-09T16:31:36.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645 2-347892350985' 2024-03-09T16:31:36.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:36.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762306 2024-03-09T16:31:36.694 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:36.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:36.695 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762306 2024-03-09T16:31:36.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:36.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-09T16:31:36.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2024-03-09T16:31:36.697 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 425201762306 2024-03-09T16:31:36.697 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:36.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 425201762306 2024-03-09T16:31:36.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:31:37.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:31:37.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:38.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-09T16:31:38.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:38.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-03-09T16:31:38.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:38.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:31:38.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-03-09T16:31:38.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:38.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-03-09T16:31:38.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-03-09T16:31:38.249 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 386547056645 2024-03-09T16:31:38.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:31:38.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056645 -lt 386547056645 2024-03-09T16:31:38.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:38.520 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350985 2024-03-09T16:31:38.521 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:38.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:31:38.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350985 2024-03-09T16:31:38.522 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:38.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350985 2024-03-09T16:31:38.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-03-09T16:31:38.523 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350985 2024-03-09T16:31:38.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:31:38.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350985 -lt 347892350985 2024-03-09T16:31:38.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:31:38.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:38.796 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:39.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:31:39.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:31:39.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:31:39.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:31:39.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:31:39.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:39.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:39.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:31:39.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:31:39.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:31:39.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T16:31:39.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:31:39.742 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:31:39.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:31:40.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:31:40.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T16:31:42.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:31:42.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:31:42.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:31:42.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:31:42.263 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:31:42.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:31:42.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:31:42.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:31:42.292 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:42.291+0000 7f8ef2c64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:42.294 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:42.293+0000 7f8ef2c64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:42.296 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:42.294+0000 7f8ef2c64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:42.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:31:42.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:31:42.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:31:42.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:42.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:42.811 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:42.809+0000 7f8ef2c64640 -1 Falling back to public interface 2024-03-09T16:31:42.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:43.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:44.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:45.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:31:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:31:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:45.242 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:45.240+0000 7f8ef2c64640 -1 osd.1 101 log_to_monitors true 2024-03-09T16:31:45.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:46.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:46.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:46.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:46.384 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:46.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:46.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [90,101) [v2:127.0.0.1:6810/1149501247,v1:127.0.0.1:6811/1149501247] [v2:127.0.0.1:6812/1149501247,v1:127.0.0.1:6813/1149501247] exists,up 46b08ab1-63ac-4b8a-aaca-7cea8f6bbb21 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:46.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:46.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:46.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:46.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:46.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:46.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:46.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:46.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:47.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:47.175 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:47.175 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:47.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:47.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:47.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:47.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-09T16:31:47.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-09T16:31:47.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309' 2024-03-09T16:31:47.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:47.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:47.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-03-09T16:31:47.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-03-09T16:31:47.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378' 2024-03-09T16:31:47.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:47.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378 2-347892350988' 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762309 2024-03-09T16:31:47.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:47.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:47.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762309 2024-03-09T16:31:47.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:47.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-09T16:31:47.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762309' 2024-03-09T16:31:47.487 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 425201762309 2024-03-09T16:31:47.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:47.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762309 2024-03-09T16:31:47.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:31:48.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:31:48.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:49.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-03-09T16:31:49.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:49.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-03-09T16:31:49.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:49.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:31:49.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-03-09T16:31:49.072 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:49.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-03-09T16:31:49.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-03-09T16:31:49.073 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 455266533378 2024-03-09T16:31:49.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:31:49.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-03-09T16:31:49.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:49.347 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-03-09T16:31:49.347 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:49.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:31:49.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-03-09T16:31:49.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:49.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-03-09T16:31:49.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-03-09T16:31:49.350 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350988 2024-03-09T16:31:49.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:31:49.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350988 -lt 347892350988 2024-03-09T16:31:49.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:31:49.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:49.623 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:49.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:31:49.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:31:49.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:31:49.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:31:49.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:31:49.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:31:49.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:31:49.967 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:31:50.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:31:50.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:31:50.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:31:50.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:31:50.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-09T16:31:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:31:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:31:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:31:51.088 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:31:51.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:31:51.465 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:31:51.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:31:51.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:31:51.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:31:51.727 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-09T16:31:51.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T16:31:51.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T16:31:51.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T16:31:51.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T16:31:51.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:31:51.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:31:51.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:31:52.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:31:54.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:31:54.095 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:31:54.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:31:54.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:31:54.097 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:31:54.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:31:54.097 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:31:54.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:31:54.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:31:54.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:31:54.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:31:54.124 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:54.123+0000 7f0060210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:54.126 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:54.125+0000 7f0060210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:54.128 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:54.126+0000 7f0060210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:54.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:31:54.378 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:31:54.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:54.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:54.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:54.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:54.658+0000 7f0060210640 -1 Falling back to public interface 2024-03-09T16:31:55.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:55.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:55.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:31:55.649 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:55.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:55.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:55.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:56.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:56.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:56.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:31:56.935 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:31:56.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:56.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:56.972 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:31:56.970+0000 7f0060210640 -1 osd.0 110 log_to_monitors true 2024-03-09T16:31:57.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:31:58.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:31:58.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:31:58.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:31:58.211 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:31:58.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:31:58.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:31:58.578 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [99,110) [v2:127.0.0.1:6802/313208335,v1:127.0.0.1:6803/313208335] [v2:127.0.0.1:6804/313208335,v1:127.0.0.1:6805/313208335] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:31:58.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:31:58.579 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:31:58.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:31:58.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:31:58.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:31:58.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:31:58.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:31:58.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:31:58.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:31:58.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:31:58.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:31:59.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:31:59.030 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:31:59.031 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:31:59.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:31:59.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:59.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:31:59.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-09T16:31:59.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-09T16:31:59.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042' 2024-03-09T16:31:59.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:59.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:31:59.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-03-09T16:31:59.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-03-09T16:31:59.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381' 2024-03-09T16:31:59.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:31:59.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:31:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350991 2024-03-09T16:31:59.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350991 2024-03-09T16:31:59.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381 2-347892350991' 2024-03-09T16:31:59.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:31:59.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-493921239042 2024-03-09T16:31:59.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:31:59.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:31:59.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-493921239042 2024-03-09T16:31:59.343 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:31:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-09T16:31:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-03-09T16:31:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 493921239042 2024-03-09T16:31:59.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:31:59.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-03-09T16:31:59.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:32:00.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:32:00.615 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:32:00.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239042 -lt 493921239042 2024-03-09T16:32:00.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:32:00.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533381 2024-03-09T16:32:00.893 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:32:00.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:32:00.895 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533381 2024-03-09T16:32:00.895 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:32:00.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-03-09T16:32:00.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533381' 2024-03-09T16:32:00.897 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 455266533381 2024-03-09T16:32:00.897 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:32:01.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533381 2024-03-09T16:32:01.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:32:01.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350991 2024-03-09T16:32:01.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:32:01.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:32:01.174 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350991 2024-03-09T16:32:01.174 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:32:01.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350991 2024-03-09T16:32:01.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350991' 2024-03-09T16:32:01.175 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350991 2024-03-09T16:32:01.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:32:01.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350991 -lt 347892350991 2024-03-09T16:32:01.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:32:01.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:32:01.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:32:01.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:32:01.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:32:02.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:32:02.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:32:02.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:32:02.090 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:32:02.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:32:02.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:32:02.427 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:32:02.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:32:02.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:32:02.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:32:02.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T16:32:04.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:32:04.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:32:04.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:32:04.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:32:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:32:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:32:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:32:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:32:04.915 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:32:04.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:32:04.942 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:04.941+0000 7f4a91411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:04.944 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:04.943+0000 7f4a91411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:04.944+0000 7f4a91411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:05.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:32:05.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:32:05.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:32:05.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:05.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:05.460 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:05.458+0000 7f4a91411640 -1 Falling back to public interface 2024-03-09T16:32:05.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:06.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:06.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:07.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:07.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:07.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:32:07.745 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:32:07.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:07.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:07.814 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:07.802+0000 7f4a91411640 -1 osd.0 117 log_to_monitors true 2024-03-09T16:32:08.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:09.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/2276953049,v1:127.0.0.1:6803/2276953049] [v2:127.0.0.1:6804/2276953049,v1:127.0.0.1:6805/2276953049] exists,up 30197b28-6ba1-4853-a4d5-366599c9a1cc 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:32:09.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:32:09.441 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:32:09.441 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:32:09.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:32:09.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:32:09.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:32:09.529 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:32:09.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:32:09.885 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:32:09.885 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:32:09.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:32:09.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:32:09.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:32:09.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-09T16:32:09.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-09T16:32:09.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-03-09T16:32:09.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:32:09.989 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:32:10.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533384 2024-03-09T16:32:10.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533384 2024-03-09T16:32:10.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533384' 2024-03-09T16:32:10.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:32:10.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350994 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350994 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533384 2-347892350994' 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-03-09T16:32:10.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:32:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:32:10.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-03-09T16:32:10.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:32:10.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-09T16:32:10.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-03-09T16:32:10.199 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 523986010114 2024-03-09T16:32:10.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:32:10.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-09T16:32:10.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:32:11.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:32:11.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:32:11.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-09T16:32:11.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:32:12.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:32:12.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:32:13.034 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-03-09T16:32:13.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:32:13.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533384 2024-03-09T16:32:13.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:32:13.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:32:13.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533384 2024-03-09T16:32:13.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:32:13.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533384 2024-03-09T16:32:13.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533384' 2024-03-09T16:32:13.038 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 455266533384 2024-03-09T16:32:13.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:32:13.308 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533384 2024-03-09T16:32:13.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:32:13.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350994 2024-03-09T16:32:13.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350994 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350994 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350994' 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.2 seq 347892350994 2024-03-09T16:32:13.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:32:13.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350995 -lt 347892350994 2024-03-09T16:32:13.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:32:13.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:32:13.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:32:13.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:32:13.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:32:13.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:32:14.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:32:14.207 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:32:14.207 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:32:14.207 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:32:14.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:32:14.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:32:14.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:32:14.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-09T16:32:14.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-09T16:32:14.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:32:14.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-09T16:32:14.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-09T16:32:14.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:14.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:15.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:15.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T16:32:15.381 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:15.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:15.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:15.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:15.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:16.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:16.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:16.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:16.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:16.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:16.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:16.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:17.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:17.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:17.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:17.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:17.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:17.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:17.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:18.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:18.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:19.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:19.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:19.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:19.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:19.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:19.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:19.226 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:19.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:19.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:20.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:20.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:20.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:20.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:20.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:20.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:20.504 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:20.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:20.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:21.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:21.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:21.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:32:21.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:21.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:32:21.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:21.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:32:22.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:32:16.034884+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:22.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:32:22.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T16:32:22.084 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:32:22.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-09T16:32:22.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:32:22.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T16:32:22.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T16:32:22.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:32:22.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:32:22.118 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:32:22.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:32:22.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:32:22.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:32:22.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:32:22.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:32:22.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:22.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:32:22.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:22.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T16:32:22.687 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-09T16:32:22.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T16:29:49.111385+0000 last_deep_scrub_stamp 2024-03-09T16:32:22.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:32:22.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:22.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:32:22.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:22.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:23.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:23.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:23.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:32:23.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:23.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:32:23.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:23.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:32:24.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:24.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:25.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:25.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:25.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:32:25.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:25.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:32:25.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:25.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:32:25.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:29:49.111385+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:25.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:32:26.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:32:26.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:32:26.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:32:26.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:32:26.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:32:26.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:32:26.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:32:26.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:32:23.195861+0000 '>' 2024-03-09T16:29:49.111385+0000 2024-03-09T16:32:26.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:32:26.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T16:32:26.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:32:26.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-09T16:32:26.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:32:26.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T16:32:26.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T16:32:26.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:32:26.878 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:32:26.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:32:26.902 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T16:32:26.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:32:26.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:32:26.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:32:26.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-09T16:32:26.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-09T16:32:26.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-09T16:32:27.285 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' removed 2024-03-09T16:32:27.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:32:27.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:32:27.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:32:27.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:32:27.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:32:27.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:32:27.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:32:27.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:32:27.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:32:27.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:32:27.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:32:27.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:32:27.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:32:27.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:32:27.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:32:27.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:32:27.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:32:27.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:32:27.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:32:27.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:32:27.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:32:27.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:32:27.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:32:27.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:32:27.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:27.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:27.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:32:27.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:32:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:32:27.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:32:27.443 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:32:27.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:32:27.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:32:27.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:32:27.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:32:27.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:32:27.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:32:27.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:32:27.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:32:27.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:32:27.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:32:27.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:32:27.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:32:27.450 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:32:27.450 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:27.450 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:27.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:32:27.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:32:27.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:32:27.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:32:27.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:32:27.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:27.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:27.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:32:27.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:32:27.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:32:27.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:27.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T16:32:28.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:32:28.029 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:28.030 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:28.030 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:32:28.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:32:28.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:32:28.108 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:32:28.109 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:32:28.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:32:28.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:29.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:32:29.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:32:29.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-03-09T16:32:29.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:32:29.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:32:29.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:32:29.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:32:29.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f819a490-4e68-47ab-858e-7bf51516a271' 2024-03-09T16:32:29.040 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:32:29.040 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:32:29.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdj+xlbqI/AxAAr1sn6afiLPJInCgTh+JIXw== 2024-03-09T16:32:29.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdj+xlbqI/AxAAr1sn6afiLPJInCgTh+JIXw=="}' 2024-03-09T16:32:29.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f819a490-4e68-47ab-858e-7bf51516a271 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:32:29.259 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:32:29.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:32:29.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdj+xlbqI/AxAAr1sn6afiLPJInCgTh+JIXw== --osd-uuid f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:32:29.298 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.297+0000 7f71ff8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:29.310 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.309+0000 7f71ff8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:29.311 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.310+0000 7f71ff8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:29.311 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.310+0000 7f71ff8c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:32:29.312 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.310+0000 7f71ff8c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:32:29.312 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.310+0000 7f71ff8c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:32:29.394 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:29.392+0000 7f71ff8c8640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:32:34.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:32:34.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:32:34.917 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:32:34.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:32:34.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:32:35.294 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:32:35.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:32:35.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:32:35.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:32:35.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:32:35.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:32:35.320 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:35.319+0000 7fef6d506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:35.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:35.321+0000 7fef6d506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:35.323 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:35.322+0000 7fef6d506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:32:35.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:32:35.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:32:35.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:35.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:35.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:35.853 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:35.849+0000 7fef6d506640 -1 Falling back to public interface 2024-03-09T16:32:36.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:36.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:36.839 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:32:36.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:32:36.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:36.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:37.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:38.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:38.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:38.116 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:32:38.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:32:38.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:38.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:38.181 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:32:38.180+0000 7fef6d506640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:32:38.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:39.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:39.441 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:32:39.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:39.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:32:39.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:39.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:39.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:40.761 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:40.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:40.762 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:32:40.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:32:40.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:40.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:41.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:42.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:42.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:42.097 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:32:42.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:32:42.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:42.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:42.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:43.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:43.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:43.377 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:32:43.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:32:43.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:43.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:43.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:44.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:44.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:44.701 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:32:44.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:32:44.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:44.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:44.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:45.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:45.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:45.984 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:32:45.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:32:45.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:45.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:46.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:47.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:47.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:47.268 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:32:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:32:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:47.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:47.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:48.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:48.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:48.551 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:32:48.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:32:48.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:48.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:49.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:49.918 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:32:49.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:49.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:32:49.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:49.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:50.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:51.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:51.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:51.478 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:32:51.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:32:51.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:51.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:51.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:52.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:52.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:52.970 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:32:52.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:32:52.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:52.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:53.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:54.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:54.595 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:32:54.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:32:54.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:54.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:55.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:56.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:56.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:56.216 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:32:56.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:32:56.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:56.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:56.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:57.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:57.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:57.825 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:32:57.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:32:57.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:57.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:58.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:32:59.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:32:59.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:32:59.216 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:32:59.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:32:59.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:32:59.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:32:59.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:00.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:00.952 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:33:00.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:00.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:33:00.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:00.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:33:01.231 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2360108786,v1:127.0.0.1:6803/2360108786] [v2:127.0.0.1:6804/2360108786,v1:127.0.0.1:6805/2360108786] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:33:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:33:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:33:01.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:33:01.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:33:01.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:33:01.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:33:01.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:33:01.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:33:01.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:33:01.241 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:33:01.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:33:01.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9' 2024-03-09T16:33:01.242 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:33:01.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9j+xlqWP1DhAAjqFRGUEfcNoY5jvvHaAAFg== 2024-03-09T16:33:01.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9j+xlqWP1DhAAjqFRGUEfcNoY5jvvHaAAFg=="}' 2024-03-09T16:33:01.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:33:01.644 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:33:01.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:33:01.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9j+xlqWP1DhAAjqFRGUEfcNoY5jvvHaAAFg== --osd-uuid 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:33:01.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.682+0000 7fa16220f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:01.720 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.719+0000 7fa16220f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:01.721 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.720+0000 7fa16220f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:01.722 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.720+0000 7fa16220f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:33:01.722 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.720+0000 7fa16220f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:33:01.722 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.720+0000 7fa16220f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:33:01.796 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:01.794+0000 7fa16220f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:33:07.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:33:07.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:33:07.550 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:33:07.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:33:07.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:33:07.903 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:33:07.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:33:07.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:33:07.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:33:07.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:33:07.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:33:07.930 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:07.928+0000 7f744400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:07.931 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:07.930+0000 7f744400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:07.932 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:07.931+0000 7f744400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:08.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:33:08.206 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:33:08.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:08.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:08.447 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:08.446+0000 7f744400c640 -1 Falling back to public interface 2024-03-09T16:33:08.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:09.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:09.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:09.483 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:33:09.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:33:09.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:09.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:09.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:10.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:10.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:10.769 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:33:10.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:33:10.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:10.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:10.800 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:33:10.799+0000 7f744400c640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:33:11.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:12.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:12.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:12.043 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:33:12.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:33:12.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:12.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:12.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:13.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:13.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:13.397 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:33:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:33:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:13.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:14.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:14.780 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:33:14.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:14.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:33:14.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:14.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:15.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:16.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:16.064 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:33:16.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:16.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:33:16.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:16.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:16.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:17.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:17.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:17.390 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:33:17.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:33:17.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:17.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:17.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:18.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:18.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:18.670 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:33:18.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:33:18.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:18.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:18.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:19.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:19.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:19.950 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:33:19.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:33:19.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:19.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:20.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:21.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:21.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:21.231 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:33:21.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:33:21.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:21.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:21.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:22.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:22.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:22.512 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:33:22.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:33:22.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:22.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:22.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:23.878 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:33:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:33:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:24.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:25.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:25.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:25.574 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:33:25.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:33:25.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:25.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:25.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:26.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:26.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:26.882 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:33:26.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:33:26.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:26.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:27.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:28.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:28.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:28.262 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:33:28.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:33:28.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:28.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:28.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:29.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:29.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:29.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:33:29.934 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:33:29.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:29.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:30.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:31.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:31.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:31.624 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:33:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:33:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:31.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:32.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:32.965 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:33:32.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:32.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:33:32.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:32.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:33.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:33:34.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:33:34.467 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:33:34.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:33:34.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:33:34.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:33:34.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:33:34.744 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/795149287,v1:127.0.0.1:6811/795149287] [v2:127.0.0.1:6812/795149287,v1:127.0.0.1:6813/795149287] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:33:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:33:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:33:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:33:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-03-09T16:33:34.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:33:35.086 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:33:35.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:33:35.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:33:35.459 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:33:35.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:33:36.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:33:38.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T16:33:38.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:33:38.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:33:38.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:33:38.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:33:38.174 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:33:38.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:33:38.445 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:33:38.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:33:38.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:33:38.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:33:38.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-03-09T16:33:38.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-03-09T16:33:38.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-03-09T16:33:38.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:33:38.548 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:33:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672963 2024-03-09T16:33:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672963 2024-03-09T16:33:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672963' 2024-03-09T16:33:38.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:33:38.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-03-09T16:33:38.651 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:33:38.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:33:38.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-03-09T16:33:38.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:33:38.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-03-09T16:33:38.654 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836490 2024-03-09T16:33:38.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-03-09T16:33:38.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:33:38.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-03-09T16:33:38.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:33:39.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:33:39.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:33:40.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-03-09T16:33:40.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:33:40.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672963 2024-03-09T16:33:40.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:33:40.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:33:40.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672963 2024-03-09T16:33:40.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:33:40.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672963 2024-03-09T16:33:40.215 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672963 2024-03-09T16:33:40.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-03-09T16:33:40.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:33:40.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672963 2024-03-09T16:33:40.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:33:40.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:33:40.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:33:40.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:33:40.850 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:33:41.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:33:41.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:33:41.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:33:41.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:33:41.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:33:41.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:33:41.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:33:41.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-03-09T16:33:41.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T16:33:41.834 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'foo' created 2024-03-09T16:33:41.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:33:42.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-03-09T16:33:42.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-09T16:33:43.306 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'csr_pool' created 2024-03-09T16:33:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:33:44.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T16:33:44.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:33:44.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:33:44.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:33:44.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:33:44.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:33:44.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:33:44.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:33:44.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:33:44.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:33:44.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:33:44.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:33:44.697 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:33:44.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:33:44.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:33:44.698 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:33:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-03-09T16:33:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-03-09T16:33:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-03-09T16:33:44.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:33:44.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:33:44.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T16:33:44.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T16:33:44.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672966' 2024-03-09T16:33:44.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:33:44.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-03-09T16:33:44.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:33:44.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:33:44.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-03-09T16:33:44.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:33:44.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-03-09T16:33:44.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-03-09T16:33:44.908 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836492 2024-03-09T16:33:44.908 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:33:45.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-09T16:33:45.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:33:46.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:33:46.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:33:46.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-03-09T16:33:46.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:33:46.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T16:33:46.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:33:46.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:33:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T16:33:46.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:33:46.465 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672966 2024-03-09T16:33:46.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T16:33:46.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T16:33:46.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:33:46.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-03-09T16:33:46.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:33:46.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:33:46.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:33:47.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:33:47.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:33:47.117 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:33:47.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:33:47.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:33:47.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:33:47.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:33:47.118 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:33:47.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:33:47.347 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:33:47.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:33:47.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:33:47.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:33:47.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:33:47.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:33:47.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:47.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:48.055 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:48.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:48.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:48.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:48.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:49.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-09T16:33:49.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:49.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:49.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:50.246 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:50.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:50.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:50.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:50.516 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-09T16:33:50.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-09T16:33:50.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:50.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:50.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:51.140 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:51.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:51.676 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:51.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:51.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:51.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:51.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-03-09T16:33:52.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-03-09T16:33:52.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:52.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:52.529 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:52.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:53.039 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:53.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-03-09T16:33:53.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-03-09T16:33:53.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:53.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:53.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:53.925 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:53.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:54.452 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:54.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:54.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:54.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:54.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-03-09T16:33:54.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-03-09T16:33:54.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:54.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:55.414 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:55.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:55.800 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:55.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:55.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:55.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:56.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-03-09T16:33:56.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-03-09T16:33:56.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:56.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:56.709 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:56.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:57.236 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:57.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:57.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:57.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:57.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-03-09T16:33:57.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:57.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:57.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:58.130 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:58.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:58.640 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:58.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:58.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:58.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:33:58.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-03-09T16:33:59.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-03-09T16:33:59.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:33:59.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T16:33:59.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:33:59.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:33:59.476 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:33:59.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:33:59.853 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:33:59.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:33:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:33:59.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:00.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-03-09T16:34:00.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-03-09T16:34:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T16:34:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-03-09T16:34:00.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-03-09T16:34:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:00.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:00.864 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:00.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:01.291 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:01.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:01.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:01.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:01.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-03-09T16:34:01.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-03-09T16:34:01.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:01.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-09T16:34:01.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:01.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:02.244 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:02.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:02.755 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:02.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:02.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:02.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:02.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-03-09T16:34:03.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-03-09T16:34:03.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:03.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:03.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:03.582 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:03.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:04.128 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:04.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:04.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:04.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:04.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-03-09T16:34:04.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:04.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:05.122 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:05.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:05.522 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:05.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:05.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:05.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:05.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-03-09T16:34:05.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:06.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:06.426 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:06.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:06.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:06.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:06.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:07.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-03-09T16:34:07.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:07.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:07.797 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:07.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:08.223 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:08.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:08.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:08.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:08.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-03-09T16:34:08.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:08.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:08.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:09.109 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:09.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:09.628 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:09.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:09.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:09.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:09.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-03-09T16:34:10.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-03-09T16:34:10.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:10.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-09T16:34:10.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-03-09T16:34:10.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:10.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:10.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-03-09T16:34:10.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:10.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:10.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:10.590 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:10.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:10.966 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:10.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:10.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:10.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:11.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-03-09T16:34:11.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-03-09T16:34:11.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:11.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:11.903 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:11.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:12.430 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:12.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:12.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:12.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:12.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-03-09T16:34:12.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-03-09T16:34:13.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:13.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:34:13.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:34:13.374 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:34:13.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:34:13.760 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:34:13.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:34:13.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:34:13.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:34:14.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-03-09T16:34:14.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-03-09T16:34:14.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-03-09T16:34:14.283 INFO:tasks.workunit.client.0.smithi146.stderr:1025+0 records in 2024-03-09T16:34:14.283 INFO:tasks.workunit.client.0.smithi146.stderr:1025+0 records out 2024-03-09T16:34:14.283 INFO:tasks.workunit.client.0.smithi146.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00186838 s, 562 MB/s 2024-03-09T16:34:14.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-03-09T16:34:14.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-03-09T16:34:14.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-09T16:34:14.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T16:34:14.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T16:34:14.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T16:34:14.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:34:14.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-09T16:34:14.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-03-09T16:34:14.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-09T16:34:14.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-09T16:34:14.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-09T16:34:14.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:34:15.020 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:15.021 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:15.021 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:34:15.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-09T16:34:15.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69596/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-09T16:34:15.095 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:34:15.096 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-09T16:34:15.096 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:34:15.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-03-09T16:34:15.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:34:15.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:34:15.104 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:34:15.104 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:15.105 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:15.105 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:34:15.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-09T16:34:15.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-09T16:34:15.183 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:34:15.183 INFO:tasks.workunit.client.0.smithi146.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-09T16:34:15.183 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:34:15.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-09T16:34:15.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T16:34:15.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:15.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:15.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:15.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:15.192 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:15.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T16:34:15.712 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T16:33:43.188361+0000 last_deep_scrub_stamp 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:15.724 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:15.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:15.725 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:15.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:33:43.188361+0000 '>' 2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:15.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:34:16.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:34:16.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:34:16.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:16.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:16.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:16.997 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:16.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:17.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:33:43.188361+0000 '>' 2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:17.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:34:18.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:34:18.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:34:18.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:18.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:18.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:18.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:18.281 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:18.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:33:43.188361+0000 '>' 2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:18.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:34:19.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:34:19.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:34:19.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:19.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:19.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:19.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:19.556 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:19.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:33:43.188361+0000 '>' 2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:19.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:34:20.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:34:20.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:34:20.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:34:20.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:34:20.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:34:20.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:34:20.840 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:34:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:33:43.188361+0000 2024-03-09T16:34:21.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:34:21.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-09T16:34:21.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:21.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-09T16:34:21.113 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-03-09T16:34:21.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:34:21.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:34:21.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-03-09T16:34:21.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-03-09T16:34:21.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:34:21.117 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:34:21.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:34:21.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:34:21.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:34:21.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:34:21.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:22.826 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-03-09T16:34:23.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:34:23.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:34:23.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:34:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:34:23.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:34:23.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:34:23.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:34:23.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:34:23.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:34:23.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:34:23.405 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:23.403+0000 7f60ac611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:23.407 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:23.406+0000 7f60ac611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:23.408 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:23.407+0000 7f60ac611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:23.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:34:23.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:34:23.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:34:23.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:23.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:23.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:23.929 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:23.928+0000 7f60ac611640 -1 Falling back to public interface 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:24.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:25.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:26.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:26.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:26.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:34:26.214 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:34:26.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:26.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:26.380 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:26.378+0000 7f60ac611640 -1 osd.1 63 log_to_monitors true 2024-03-09T16:34:26.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:27.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/4116686368,v1:127.0.0.1:6811/4116686368] [v2:127.0.0.1:6812/4116686368,v1:127.0.0.1:6813/4116686368] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:34:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:34:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:34:27.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:34:27.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:34:27.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:34:27.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:34:27.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:34:27.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:34:27.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:34:27.965 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:34:28.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:34:28.237 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:34:28.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:34:28.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:28.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:34:28.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T16:34:28.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T16:34:28.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T16:34:28.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:28.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:34:28.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-09T16:34:28.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-09T16:34:28.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-287762808834' 2024-03-09T16:34:28.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:28.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T16:34:28.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:28.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:34:28.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T16:34:28.447 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:28.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T16:34:28.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T16:34:28.449 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836502 2024-03-09T16:34:28.449 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:28.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T16:34:28.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:34:29.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:34:29.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:29.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-09T16:34:29.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:29.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-09T16:34:29.998 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:29.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:34:29.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-09T16:34:29.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:30.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-09T16:34:30.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-09T16:34:30.001 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 287762808834 2024-03-09T16:34:30.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:34:30.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-09T16:34:30.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:34:30.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:30.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:30.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:34:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:34:30.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:34:30.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:34:30.637 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:34:30.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:34:30.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:34:30.638 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:34:30.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:34:30.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:34:30.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:30.954 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-09T16:34:31.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-03-09T16:34:31.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:34:31.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:34:31.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:34:31.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:34:31.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:34:31.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:34:31.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:34:31.602 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:34:31.603 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:34:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:34:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:34:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:34:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:34:33.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:34:33.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:34:33.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:34:33.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:34:33.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:34:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:33.570+0000 7f463d424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:33.574 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:33.572+0000 7f463d424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:33.575 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:33.573+0000 7f463d424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:33.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:34.094 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:34.092+0000 7f463d424640 -1 Falling back to public interface 2024-03-09T16:34:34.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:35.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:35.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:35.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:34:35.100 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:34:35.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:35.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:35.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:36.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:36.530 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:36.528+0000 7f463d424640 -1 osd.0 68 log_to_monitors true 2024-03-09T16:34:36.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:37.726 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:38.002 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/2967220615,v1:127.0.0.1:6803/2967220615] [v2:127.0.0.1:6804/2967220615,v1:127.0.0.1:6805/2967220615] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:34:38.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:34:38.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:34:38.003 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:34:38.091 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:34:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:34:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:34:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:34:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:38.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:34:38.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-09T16:34:38.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-09T16:34:38.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314' 2024-03-09T16:34:38.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:38.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:34:38.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-03-09T16:34:38.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-03-09T16:34:38.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314 1-287762808837' 2024-03-09T16:34:38.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:38.571 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645314 2024-03-09T16:34:38.571 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:34:38.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645314 2024-03-09T16:34:38.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:38.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-09T16:34:38.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-03-09T16:34:38.574 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 309237645314 2024-03-09T16:34:38.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:38.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 309237645314 2024-03-09T16:34:38.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:34:39.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:34:39.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:40.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-03-09T16:34:40.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:40.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808837 2024-03-09T16:34:40.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:34:40.144 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808837 2024-03-09T16:34:40.144 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-03-09T16:34:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808837' 2024-03-09T16:34:40.145 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 287762808837 2024-03-09T16:34:40.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:34:40.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-03-09T16:34:40.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:34:40.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:40.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:34:40.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:34:40.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:34:40.761 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:34:41.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:34:41.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:34:41.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:41.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:41.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:34:41.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:34:41.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:34:41.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:41.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-09T16:34:41.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:34:41.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:34:41.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:34:41.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-03-09T16:34:42.881 INFO:tasks.workunit.client.0.smithi146.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-03-09T16:34:43.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:34:43.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:34:43.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:34:43.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:34:43.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:34:43.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:34:43.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:34:43.402 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:34:43.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:34:43.429 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:43.428+0000 7f5c8dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:43.431 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:43.430+0000 7f5c8dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:43.432 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:43.431+0000 7f5c8dc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:43.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:43.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:43.948+0000 7f5c8dc13640 -1 Falling back to public interface 2024-03-09T16:34:43.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:44.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:45.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:46.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:46.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:46.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:34:46.237 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:34:46.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:46.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:46.487 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:46.486+0000 7f5c8dc13640 -1 osd.1 73 log_to_monitors true 2024-03-09T16:34:46.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:47.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:47.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:47.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:34:47.513 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:34:47.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:47.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [67,73) [v2:127.0.0.1:6810/623131111,v1:127.0.0.1:6811/623131111] [v2:127.0.0.1:6812/623131111,v1:127.0.0.1:6813/623131111] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:34:47.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:34:47.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:34:47.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:34:47.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:34:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:34:47.820 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:34:47.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:34:47.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:34:47.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:34:47.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:34:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:34:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:34:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:34:48.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:48.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:34:48.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645317 2024-03-09T16:34:48.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645317 2024-03-09T16:34:48.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317' 2024-03-09T16:34:48.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:48.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645317 2024-03-09T16:34:48.379 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:48.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:34:48.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645317 2024-03-09T16:34:48.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:48.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645317 2024-03-09T16:34:48.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2024-03-09T16:34:48.382 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 309237645317 2024-03-09T16:34:48.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-03-09T16:34:48.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:34:49.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:34:49.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:49.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645317 -lt 309237645317 2024-03-09T16:34:49.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:49.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-09T16:34:49.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:49.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:34:49.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-09T16:34:49.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T16:34:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-09T16:34:49.940 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 330712481794 2024-03-09T16:34:49.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:34:50.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-09T16:34:50.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:34:50.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:50.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:50.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:34:50.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:34:50.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:34:50.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:34:50.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:34:50.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:34:51.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:34:51.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:34:51.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:34:51.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:34:51.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-09T16:34:51.233 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-03-09T16:34:51.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:34:51.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:34:51.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:34:51.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:34:51.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:34:51.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T16:34:53.310 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:34:53.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:34:53.334 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:34:53.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:34:53.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:34:53.338 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:53.337+0000 7f3b4ce13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:53.341 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:53.339+0000 7f3b4ce13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:53.341 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:53.340+0000 7f3b4ce13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:34:53.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:34:53.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:34:53.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:34:53.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:53.596 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:53.859 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:53.858+0000 7f3b4ce13640 -1 Falling back to public interface 2024-03-09T16:34:53.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:54.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:54.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:54.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:34:54.866 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:34:54.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:54.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:55.165 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:56.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:56.336 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:34:56.334+0000 7f3b4ce13640 -1 osd.0 78 log_to_monitors true 2024-03-09T16:34:56.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:34:57.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:34:57.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:34:57.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:34:57.479 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:34:57.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:34:57.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/1103542329,v1:127.0.0.1:6803/1103542329] [v2:127.0.0.1:6804/1103542329,v1:127.0.0.1:6805/1103542329] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:34:57.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:34:57.787 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:34:57.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:34:57.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:34:57.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:34:57.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:34:57.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:34:57.876 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:34:58.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:34:58.149 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:34:58.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:34:58.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:58.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:34:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-09T16:34:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-09T16:34:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-03-09T16:34:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:34:58.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:34:58.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-03-09T16:34:58.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-03-09T16:34:58.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-03-09T16:34:58.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:58.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-03-09T16:34:58.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:58.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:34:58.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-03-09T16:34:58.387 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:58.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-09T16:34:58.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-03-09T16:34:58.389 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 352187318274 2024-03-09T16:34:58.389 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:58.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-03-09T16:34:58.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:34:59.660 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:34:59.661 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:34:59.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-09T16:34:59.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:34:59.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-03-09T16:34:59.941 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:34:59.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:34:59.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-03-09T16:34:59.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:34:59.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-03-09T16:34:59.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-03-09T16:34:59.944 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 330712481797 2024-03-09T16:34:59.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:00.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-03-09T16:35:00.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:00.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:00.238 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:00.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:00.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:00.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:00.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:00.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-09T16:35:01.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:01.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:01.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:35:01.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:01.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T16:35:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:35:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:03.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:03.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:03.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:03.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:35:03.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:03.386 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:35:03.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:35:03.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:03.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:03.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:03.411 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:03.409+0000 7f83fa616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:03.413 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:03.411+0000 7f83fa616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:03.414 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:03.413+0000 7f83fa616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:03.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:35:03.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:03.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:35:03.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:03.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:03.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:03.935 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:03.933+0000 7f83fa616640 -1 Falling back to public interface 2024-03-09T16:35:03.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:04.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:05.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:06.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:06.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:06.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:06.217 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:06.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:06.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:06.308 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:06.306+0000 7f83fa616640 -1 osd.1 84 log_to_monitors true 2024-03-09T16:35:06.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:07.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/1400973488,v1:127.0.0.1:6811/1400973488] [v2:127.0.0.1:6812/1400973488,v1:127.0.0.1:6813/1400973488] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:07.833 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:07.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:07.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:07.835 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:07.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:07.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:07.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:07.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:07.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:07.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:08.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:08.219 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:08.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:08.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:08.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:08.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-03-09T16:35:08.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-03-09T16:35:08.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-03-09T16:35:08.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:08.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:08.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-09T16:35:08.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-09T16:35:08.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-03-09T16:35:08.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:08.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-03-09T16:35:08.426 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:08.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:08.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-03-09T16:35:08.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:08.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-03-09T16:35:08.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-03-09T16:35:08.430 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 352187318277 2024-03-09T16:35:08.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:08.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-03-09T16:35:08.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:09.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:09.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:09.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-03-09T16:35:09.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:09.982 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-03-09T16:35:09.982 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:09.983 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:09.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-03-09T16:35:09.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:09.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-09T16:35:09.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-03-09T16:35:09.986 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 377957122050 2024-03-09T16:35:09.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:10.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-09T16:35:10.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:10.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:10.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:10.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:10.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:10.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:10.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:10.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:10.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:10.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:10.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:10.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:10.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:10.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:10.896 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-09T16:35:11.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:11.240 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:35:11.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:11.241 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:11.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:11.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:11.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:11.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:11.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T16:35:11.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:11.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:35:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:35:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T16:35:13.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:35:13.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:35:13.368 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:35:13.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:13.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:35:13.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:35:13.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:13.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:13.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:13.395 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:13.393+0000 7f7d41306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:13.396+0000 7f7d41306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:13.399 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:13.397+0000 7f7d41306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:13.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:35:13.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:13.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:35:13.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:13.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:13.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:13.921 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:13.920+0000 7f7d41306640 -1 Falling back to public interface 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:14.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:15.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:16.389 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:16.387+0000 7f7d41306640 -1 osd.0 89 log_to_monitors true 2024-03-09T16:35:16.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:17.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:17.801 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/393386311,v1:127.0.0.1:6803/393386311] [v2:127.0.0.1:6804/393386311,v1:127.0.0.1:6805/393386311] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:35:17.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:17.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:17.802 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:17.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:17.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:17.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:17.890 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:18.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:18.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:18.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:18.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-09T16:35:18.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-09T16:35:18.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-03-09T16:35:18.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:18.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-03-09T16:35:18.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:18.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:18.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-03-09T16:35:18.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:18.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-09T16:35:18.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-03-09T16:35:18.515 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 399431958530 2024-03-09T16:35:18.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:18.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-03-09T16:35:18.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:19.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:19.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:20.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-03-09T16:35:20.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:20.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-03-09T16:35:20.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:20.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:20.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-03-09T16:35:20.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:20.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-03-09T16:35:20.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-03-09T16:35:20.070 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 377957122053 2024-03-09T16:35:20.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:20.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-03-09T16:35:20.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:20.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:20.342 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:20.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:20.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:20.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:20.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:20.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:20.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:20.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:21.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:21.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:21.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:21.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-03-09T16:35:21.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:35:21.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-09T16:35:21.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:35:21.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:21.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:21.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T16:35:23.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:35:23.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:23.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:23.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:23.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:23.321 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:23.319+0000 7f7afd6c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:23.323 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:23.322+0000 7f7afd6c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:23.324 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:23.323+0000 7f7afd6c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:23.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:35:23.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:23.841 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:23.840+0000 7f7afd6c8640 -1 Falling back to public interface 2024-03-09T16:35:23.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:24.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:26.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:26.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:26.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:26.128 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:26.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:26.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:26.325 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:26.323+0000 7f7afd6c8640 -1 osd.1 95 log_to_monitors true 2024-03-09T16:35:26.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:27.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:27.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:27.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:27.440 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:27.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:27.730 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/2447502964,v1:127.0.0.1:6811/2447502964] [v2:127.0.0.1:6812/2447502964,v1:127.0.0.1:6813/2447502964] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:35:27.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:27.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:27.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:27.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:28.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:28.146 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:28.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:28.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:28.146 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-03-09T16:35:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-03-09T16:35:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-03-09T16:35:28.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:28.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:28.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-09T16:35:28.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-09T16:35:28.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-03-09T16:35:28.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:28.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-03-09T16:35:28.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:28.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:28.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-03-09T16:35:28.395 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:28.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-03-09T16:35:28.396 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-03-09T16:35:28.397 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 399431958533 2024-03-09T16:35:28.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:28.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-03-09T16:35:28.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:29.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:29.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:29.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-03-09T16:35:29.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:29.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-03-09T16:35:29.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:29.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:29.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-03-09T16:35:29.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:29.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-09T16:35:29.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-03-09T16:35:29.950 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 425201762306 2024-03-09T16:35:29.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:30.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-09T16:35:30.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:30.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:30.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:30.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:30.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:30.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:30.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:30.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:31.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:31.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:31.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-03-09T16:35:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:35:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-09T16:35:31.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-03-09T16:35:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:35:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:35:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-03-09T16:35:31.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:31.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:31.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:35:31.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:35:33.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:33.349 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:35:33.350 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:35:33.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:33.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:35:33.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:35:33.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:33.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:33.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:33.378 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:33.376+0000 7f618c4f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:33.379+0000 7f618c4f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:33.380+0000 7f618c4f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:33.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:35:33.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:33.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:35:33.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:33.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:33.902 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:33.901+0000 7f618c4f7640 -1 Falling back to public interface 2024-03-09T16:35:33.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:34.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:34.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:34.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:35.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:36.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:36.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:36.268+0000 7f618c4f7640 -1 osd.0 101 log_to_monitors true 2024-03-09T16:35:36.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:37.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:37.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:37.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:37.501 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:37.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:37.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:35:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/188549512,v1:127.0.0.1:6803/188549512] [v2:127.0.0.1:6804/188549512,v1:127.0.0.1:6805/188549512] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:35:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:37.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:37.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:37.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:37.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:37.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:37.884 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:38.152 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:38.153 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:38.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:38.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:38.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:38.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-09T16:35:38.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-09T16:35:38.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-03-09T16:35:38.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:38.257 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-03-09T16:35:38.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:38.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:38.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-03-09T16:35:38.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:38.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-09T16:35:38.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-03-09T16:35:38.405 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 450971566082 2024-03-09T16:35:38.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:38.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-03-09T16:35:38.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:39.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:39.678 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:39.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-03-09T16:35:39.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:39.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-03-09T16:35:39.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:39.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:39.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-03-09T16:35:39.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:39.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-09T16:35:39.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-03-09T16:35:39.964 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 425201762309 2024-03-09T16:35:39.964 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:40.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-03-09T16:35:40.255 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:40.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:40.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:40.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:40.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:40.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:40.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:40.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:40.869 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:41.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:35:41.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:41.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:35:41.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:41.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:41.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T16:35:43.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:35:43.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:43.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:43.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:43.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:43.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:43.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:43.242 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:43.239+0000 7f2ed8e8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:43.244 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:43.242+0000 7f2ed8e8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:43.245 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:43.243+0000 7f2ed8e8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:43.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:43.759 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:43.757+0000 7f2ed8e8b640 -1 Falling back to public interface 2024-03-09T16:35:43.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:44.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:44.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:45.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:46.049 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:46.081 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:46.079+0000 7f2ed8e8b640 -1 osd.1 106 log_to_monitors true 2024-03-09T16:35:46.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:47.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3566309384,v1:127.0.0.1:6811/3566309384] [v2:127.0.0.1:6812/3566309384,v1:127.0.0.1:6813/3566309384] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:47.657 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:47.658 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:47.658 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:47.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:47.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:47.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:47.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:48.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:48.021 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:48.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:48.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:48.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:48.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-03-09T16:35:48.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-03-09T16:35:48.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-03-09T16:35:48.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:48.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:48.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-09T16:35:48.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-09T16:35:48.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-03-09T16:35:48.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:48.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-03-09T16:35:48.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:48.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:48.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-03-09T16:35:48.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:48.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-03-09T16:35:48.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-03-09T16:35:48.233 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 450971566085 2024-03-09T16:35:48.233 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:48.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-03-09T16:35:48.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:49.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:49.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:49.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-03-09T16:35:49.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:49.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-03-09T16:35:49.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:49.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:49.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-03-09T16:35:49.790 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:49.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-09T16:35:49.791 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-03-09T16:35:49.791 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 472446402562 2024-03-09T16:35:49.792 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:35:50.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-03-09T16:35:50.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:35:50.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:50.063 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:50.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:35:50.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:35:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:35:50.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:35:50.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:35:50.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:35:50.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:35:51.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:35:51.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:35:51.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:35:51.008 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:35:51.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:35:51.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:35:51.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:51.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T16:35:53.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:35:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:35:53.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:35:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:35:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:35:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:35:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:35:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:35:53.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:35:53.148 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:35:53.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:35:53.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:35:53.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:35:53.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:35:53.173 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:53.172+0000 7f9e052c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:53.175 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:53.174+0000 7f9e052c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:53.176 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:53.175+0000 7f9e052c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:53.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:53.690 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:53.689+0000 7f9e052c8640 -1 Falling back to public interface 2024-03-09T16:35:53.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:54.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:54.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:55.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:56.026 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:35:56.024+0000 7f9e052c8640 -1 osd.1 112 log_to_monitors true 2024-03-09T16:35:56.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:35:57.336 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:35:57.685 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1340837679,v1:127.0.0.1:6811/1340837679] [v2:127.0.0.1:6812/1340837679,v1:127.0.0.1:6813/1340837679] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:35:57.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:35:57.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:35:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:35:57.687 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:35:57.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:35:57.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:35:57.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:35:57.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:35:57.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:35:57.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:35:57.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:35:57.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:35:57.775 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:35:58.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:35:58.097 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:35:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:35:58.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:58.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:35:58.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-03-09T16:35:58.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-03-09T16:35:58.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-03-09T16:35:58.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:35:58.231 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:35:58.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-09T16:35:58.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-09T16:35:58.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-03-09T16:35:58.334 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:58.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-03-09T16:35:58.334 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:58.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:35:58.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-03-09T16:35:58.336 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:58.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-03-09T16:35:58.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-03-09T16:35:58.337 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 450971566088 2024-03-09T16:35:58.337 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:58.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566087 -lt 450971566088 2024-03-09T16:35:58.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:35:59.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:35:59.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:35:59.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-03-09T16:35:59.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:35:59.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-03-09T16:35:59.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:35:59.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:35:59.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-03-09T16:35:59.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:35:59.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-09T16:35:59.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-03-09T16:35:59.889 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 498216206338 2024-03-09T16:35:59.889 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:00.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-09T16:36:00.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:00.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:00.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:00.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:00.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:00.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:00.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:00.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:00.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:00.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:00.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:00.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:00.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:00.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:00.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:01.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:01.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:01.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:01.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T16:36:01.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:01.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T16:36:01.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:01.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:01.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:01.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:01.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:03.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:03.472 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:36:03.473 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:36:03.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:03.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:36:03.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:36:03.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:03.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:03.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:03.500 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:03.499+0000 7f356bec4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:03.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:03.501+0000 7f356bec4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:03.503 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:03.502+0000 7f356bec4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:03.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:04.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:04.024 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:04.023+0000 7f356bec4640 -1 Falling back to public interface 2024-03-09T16:36:05.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:05.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:05.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:05.025 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:05.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:05.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:05.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:06.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:06.395 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:06.394+0000 7f356bec4640 -1 osd.1 117 log_to_monitors true 2024-03-09T16:36:06.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:07.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:08.128 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/225991740,v1:127.0.0.1:6811/225991740] [v2:127.0.0.1:6812/225991740,v1:127.0.0.1:6813/225991740] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:36:08.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:08.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:08.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:08.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:08.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:08.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:08.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:08.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-03-09T16:36:08.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-03-09T16:36:08.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-03-09T16:36:08.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:08.714 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:08.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-03-09T16:36:08.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-03-09T16:36:08.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-03-09T16:36:08.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:08.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-03-09T16:36:08.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:08.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:08.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-03-09T16:36:08.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:08.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-03-09T16:36:08.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-03-09T16:36:08.860 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 450971566091 2024-03-09T16:36:08.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:09.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-03-09T16:36:09.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:10.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:36:10.135 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:10.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566091 2024-03-09T16:36:10.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:10.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-03-09T16:36:10.418 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:10.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:10.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-03-09T16:36:10.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:10.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-03-09T16:36:10.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-03-09T16:36:10.421 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 519691042818 2024-03-09T16:36:10.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-03-09T16:36:10.693 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:10.693 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:10.693 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:11.035 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:11.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:11.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:11.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:11.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:11.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:11.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:11.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:11.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-03-09T16:36:11.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-09T16:36:11.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:36:11.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:11.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:11.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:11.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:12.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:12.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:36:13.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:13.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:13.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:36:13.874 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:13.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:13.901 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:13.900+0000 7f809c40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:13.903 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:13.902+0000 7f809c40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:13.905 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:13.903+0000 7f809c40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:14.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:14.426 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:14.425+0000 7f809c40f640 -1 Falling back to public interface 2024-03-09T16:36:14.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:15.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:15.714 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:16.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:16.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:16.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:16.716 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:16.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:16.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:16.737 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:16.735+0000 7f809c40f640 -1 osd.1 123 log_to_monitors true 2024-03-09T16:36:17.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/2096420295,v1:127.0.0.1:6811/2096420295] [v2:127.0.0.1:6812/2096420295,v1:127.0.0.1:6813/2096420295] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:18.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:18.356 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:18.356 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:18.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:18.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:18.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:18.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:18.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:18.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:18.444 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:18.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:18.445 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:18.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:18.753 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:18.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:18.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:18.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:18.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-03-09T16:36:18.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-03-09T16:36:18.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-03-09T16:36:18.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:18.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:18.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-03-09T16:36:18.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-03-09T16:36:18.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-03-09T16:36:18.959 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:18.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-03-09T16:36:18.960 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:18.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:18.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-03-09T16:36:18.961 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:18.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-03-09T16:36:18.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-03-09T16:36:18.962 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 450971566094 2024-03-09T16:36:18.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:19.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-03-09T16:36:19.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:19.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-03-09T16:36:19.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:19.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:19.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-03-09T16:36:19.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:19.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-03-09T16:36:19.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-03-09T16:36:19.257 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 545460846594 2024-03-09T16:36:19.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:19.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-03-09T16:36:19.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:19.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:19.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:19.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:20.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:20.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:20.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:20.143 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:20.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:20.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:20.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:20.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-09T16:36:20.483 INFO:tasks.workunit.client.0.smithi146.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-09T16:36:20.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:20.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T16:36:20.483 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-03-09T16:36:20.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:36:20.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:20.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:20.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-09T16:36:20.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-03-09T16:36:20.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:36:20.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:20.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:20.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-09T16:36:20.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-03-09T16:36:20.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:36:20.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:20.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:20.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-09T16:36:20.487 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-03-09T16:36:20.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T16:36:20.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:20.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:36:20.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:20.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:36:20.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T16:36:22.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:36:22.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:22.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:22.433 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:22.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:22.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:22.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:22.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:22.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:22.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:36:22.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:22.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:22.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:22.463 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:22.461+0000 7fcfa1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:22.465 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:22.463+0000 7fcfa1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:22.466 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:22.465+0000 7fcfa1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:22.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:36:22.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:22.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:22.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:22.979 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:22.978+0000 7fcfa1e0c640 -1 Falling back to public interface 2024-03-09T16:36:22.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:23.990 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:23.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:23.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:24.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:25.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:25.397 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:25.396+0000 7fcfa1e0c640 -1 osd.0 129 log_to_monitors true 2024-03-09T16:36:25.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:26.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:26.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:26.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:26.550 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:26.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:26.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/1359962806,v1:127.0.0.1:6803/1359962806] [v2:127.0.0.1:6804/1359962806,v1:127.0.0.1:6805/1359962806] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:26.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:26.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:26.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:26.974 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:26.974 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:26.974 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:26.975 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:26.975 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:27.062 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:27.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:27.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:27.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-03-09T16:36:27.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-03-09T16:36:27.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-03-09T16:36:27.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:27.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:27.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-03-09T16:36:27.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-03-09T16:36:27.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-03-09T16:36:27.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:27.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-03-09T16:36:27.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:27.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:27.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-03-09T16:36:27.564 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:27.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-03-09T16:36:27.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-03-09T16:36:27.566 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 571230650370 2024-03-09T16:36:27.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:27.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-03-09T16:36:27.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:28.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:36:28.838 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-03-09T16:36:29.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:30.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:36:30.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:30.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-03-09T16:36:30.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:30.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-03-09T16:36:30.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:30.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:30.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-03-09T16:36:30.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:30.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-03-09T16:36:30.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-03-09T16:36:30.405 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 545460846597 2024-03-09T16:36:30.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:30.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846597 -lt 545460846597 2024-03-09T16:36:30.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:30.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:30.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:31.014 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:31.015 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:31.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:31.286 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:31.286 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:31.286 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:31.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:36:31.627 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:31.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:31.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:31.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:31.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:31.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:33.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:33.569 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:33.570 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:33.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:36:33.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:36:33.571 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:33.597 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:33.595+0000 7f42d3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:33.599 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:33.597+0000 7f42d3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:33.600 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:33.599+0000 7f42d3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:33.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:33.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:34.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:34.122 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:34.121+0000 7f42d3e0c640 -1 Falling back to public interface 2024-03-09T16:36:35.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:35.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:35.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:35.121 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:35.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:35.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:35.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:36.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:36.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:36.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:36.400 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:36.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:36.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:36.502 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:36.501+0000 7f42d3e0c640 -1 osd.1 134 log_to_monitors true 2024-03-09T16:36:36.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:37.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:37.998 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/2069932356,v1:127.0.0.1:6811/2069932356] [v2:127.0.0.1:6812/2069932356,v1:127.0.0.1:6813/2069932356] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:36:37.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:37.999 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:38.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:38.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:38.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:38.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:38.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:38.090 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:38.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:38.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:38.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:38.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-03-09T16:36:38.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-03-09T16:36:38.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-03-09T16:36:38.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:38.468 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-03-09T16:36:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-03-09T16:36:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-03-09T16:36:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:38.573 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-03-09T16:36:38.573 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:38.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:38.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-03-09T16:36:38.574 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-03-09T16:36:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-03-09T16:36:38.575 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 571230650373 2024-03-09T16:36:38.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:38.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-03-09T16:36:38.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:39.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:36:39.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:40.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-03-09T16:36:40.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:40.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-03-09T16:36:40.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:40.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:40.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-03-09T16:36:40.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:40.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-03-09T16:36:40.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-03-09T16:36:40.126 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 592705486850 2024-03-09T16:36:40.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:40.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-03-09T16:36:40.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:40.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:40.416 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:40.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:40.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:40.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:40.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:40.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:40.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:40.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:40.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:41.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:41.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:41.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:41.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-09T16:36:41.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:36:41.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:41.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:41.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:41.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T16:36:41.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:41.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:41.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:36:41.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:41.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:41.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-03-09T16:36:42.796 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:36:43.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:36:43.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:43.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:36:43.317 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:43.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:43.345 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:43.343+0000 7f95a60b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:43.347 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:43.345+0000 7f95a60b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:43.347 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:43.346+0000 7f95a60b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:43.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:43.863 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:43.862+0000 7f95a60b3640 -1 Falling back to public interface 2024-03-09T16:36:43.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:44.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:44.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:44.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:44.871 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:44.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:44.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:45.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:46.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:46.244 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:46.242+0000 7f95a60b3640 -1 osd.1 140 log_to_monitors true 2024-03-09T16:36:46.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:47.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:36:47.769 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/4142353653,v1:127.0.0.1:6811/4142353653] [v2:127.0.0.1:6812/4142353653,v1:127.0.0.1:6813/4142353653] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:47.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:47.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:47.771 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:47.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:47.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:47.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:47.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:48.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:48.129 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:48.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:48.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:48.129 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:48.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-03-09T16:36:48.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-03-09T16:36:48.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-03-09T16:36:48.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:48.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-03-09T16:36:48.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:48.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:48.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-03-09T16:36:48.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:48.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-03-09T16:36:48.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-03-09T16:36:48.360 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 571230650376 2024-03-09T16:36:48.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:48.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-03-09T16:36:48.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:49.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:36:49.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:49.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650376 -lt 571230650376 2024-03-09T16:36:49.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:49.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-03-09T16:36:49.905 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:49.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:49.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-03-09T16:36:49.907 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:49.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-03-09T16:36:49.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-03-09T16:36:49.909 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 618475290626 2024-03-09T16:36:49.909 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:36:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-03-09T16:36:50.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:36:50.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:50.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:50.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:36:50.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:36:50.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:36:50.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:36:50.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:36:50.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:36:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:36:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:36:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:36:51.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:36:51.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-09T16:36:51.157 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:36:51.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:36:51.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:36:51.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:36:51.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:36:51.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-03-09T16:36:52.661 INFO:tasks.workunit.client.0.smithi146.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:36:53.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:36:53.181 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:36:53.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:36:53.183 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:36:53.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:36:53.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:36:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:36:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:36:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:36:53.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:36:53.210 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:53.208+0000 7fc0cae17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:53.212 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:53.210+0000 7fc0cae17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:53.214 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:53.211+0000 7fc0cae17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:36:53.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:36:53.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:36:53.465 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:36:53.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:53.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:53.728 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:53.726+0000 7fc0cae17640 -1 Falling back to public interface 2024-03-09T16:36:53.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:54.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:55.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:56.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:56.051 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:36:56.048+0000 7fc0cae17640 -1 osd.0 145 log_to_monitors true 2024-03-09T16:36:56.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:36:57.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:36:57.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:36:57.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:36:57.307 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:36:57.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:36:57.307 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/3009975405,v1:127.0.0.1:6803/3009975405] [v2:127.0.0.1:6804/3009975405,v1:127.0.0.1:6805/3009975405] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:36:57.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:36:57.584 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:36:57.584 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:36:57.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:36:57.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:36:57.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:36:57.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:36:57.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:36:57.942 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:36:57.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:36:57.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:57.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:36:58.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-03-09T16:36:58.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-03-09T16:36:58.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-03-09T16:36:58.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:36:58.046 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:36:58.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:58.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:36:58.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-03-09T16:36:58.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:58.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-03-09T16:36:58.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-03-09T16:36:58.153 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 639950127106 2024-03-09T16:36:58.153 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:58.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 639950127106 2024-03-09T16:36:58.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:36:59.450 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:36:59.450 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:36:59.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-03-09T16:36:59.727 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:36:59.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-03-09T16:36:59.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:36:59.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:36:59.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-03-09T16:36:59.730 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:36:59.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-03-09T16:36:59.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-03-09T16:36:59.731 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 618475290629 2024-03-09T16:36:59.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:00.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-03-09T16:37:00.000 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:00.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:00.001 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:00.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:00.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:00.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:00.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:00.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:00.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:00.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:00.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:00.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:01.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T16:37:02.423 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:37:02.423 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:02.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:02.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:37:02.944 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:37:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:02.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:37:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:37:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:02.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:02.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:02.971 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:02.968+0000 7fd2b1213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:02.973 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:02.971+0000 7fd2b1213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:02.974 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:02.972+0000 7fd2b1213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:03.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:03.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:03.493 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:03.492+0000 7fd2b1213640 -1 Falling back to public interface 2024-03-09T16:37:03.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:04.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:04.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:05.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:05.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:05.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:37:05.806 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:37:05.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:05.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:05.899 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:05.898+0000 7fd2b1213640 -1 osd.1 150 log_to_monitors true 2024-03-09T16:37:06.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:07.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:07.592 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/353282579,v1:127.0.0.1:6811/353282579] [v2:127.0.0.1:6812/353282579,v1:127.0.0.1:6813/353282579] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:37:07.593 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:37:07.594 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:37:07.594 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:37:07.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:37:07.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:37:07.594 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:37:07.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:37:07.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:37:07.952 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:37:07.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:37:07.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:07.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:37:08.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-03-09T16:37:08.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-03-09T16:37:08.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-03-09T16:37:08.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:08.069 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:37:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-03-09T16:37:08.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-03-09T16:37:08.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-03-09T16:37:08.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:08.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-03-09T16:37:08.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:08.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:37:08.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-03-09T16:37:08.175 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:08.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-03-09T16:37:08.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-03-09T16:37:08.176 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 639950127109 2024-03-09T16:37:08.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:08.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127109 2024-03-09T16:37:08.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:37:09.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:37:09.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127109 -lt 639950127109 2024-03-09T16:37:09.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:09.734 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-03-09T16:37:09.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:09.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:37:09.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-03-09T16:37:09.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:09.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-03-09T16:37:09.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-03-09T16:37:09.738 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 661424963586 2024-03-09T16:37:09.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:10.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-03-09T16:37:10.010 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:10.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:10.013 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:10.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:10.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:10.634 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:10.634 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:10.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-09T16:37:10.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:10.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:37:11.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:11.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:37:11.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:12.434 INFO:tasks.workunit.client.0.smithi146.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:13.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:13.029 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:13.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:37:13.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:13.032 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:37:13.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:37:13.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:13.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:13.032 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:13.056+0000 7fc94d738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:13.060 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:13.058+0000 7fc94d738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:13.061 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:13.060+0000 7fc94d738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:37:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:13.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:13.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:13.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:13.588 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:13.587+0000 7fc94d738640 -1 Falling back to public interface 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:14.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:14.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:15.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:15.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:15.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:37:15.868 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:37:15.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:15.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:15.874 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:15.872+0000 7fc94d738640 -1 osd.0 156 log_to_monitors true 2024-03-09T16:37:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:17.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:17.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:17.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:37:17.182 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:37:17.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:17.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:17.492 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1183165874,v1:127.0.0.1:6803/1183165874] [v2:127.0.0.1:6804/1183165874,v1:127.0.0.1:6805/1183165874] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:37:17.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:37:17.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:37:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:37:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:37:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:37:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:37:17.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:37:17.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:37:17.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:37:17.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:37:17.993 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:37:17.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:37:17.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:17.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:37:18.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-03-09T16:37:18.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-03-09T16:37:18.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-03-09T16:37:18.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:18.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:37:18.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-03-09T16:37:18.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-03-09T16:37:18.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-03-09T16:37:18.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:18.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-03-09T16:37:18.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:18.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:37:18.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-03-09T16:37:18.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:18.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-03-09T16:37:18.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-03-09T16:37:18.216 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 687194767362 2024-03-09T16:37:18.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:18.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 687194767362 2024-03-09T16:37:18.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:37:19.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:37:19.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:19.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-03-09T16:37:19.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:19.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-03-09T16:37:19.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:19.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:37:19.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-03-09T16:37:19.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:19.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-03-09T16:37:19.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-03-09T16:37:19.787 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 661424963589 2024-03-09T16:37:19.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:20.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-03-09T16:37:20.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:20.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:20.065 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:20.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:20.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:20.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:20.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:20.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:20.669 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:20.670 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:37:21.005 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:21.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:21.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T16:37:22.231 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:37:22.232 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:37:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:37:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:37:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:22.822 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:37:22.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:37:22.824 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:37:22.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:22.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:37:22.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:37:22.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:22.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:22.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:22.851 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:22.849+0000 7f781be23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:22.853 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:22.851+0000 7f781be23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:22.854 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:22.853+0000 7f781be23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:23.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:37:23.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:23.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:23.368 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:23.367+0000 7f781be23640 -1 Falling back to public interface 2024-03-09T16:37:23.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:24.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:24.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:25.453 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:25.452+0000 7f781be23640 -1 osd.1 162 log_to_monitors true 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:25.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:26.107 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 166 up_thru 160 down_at 163 last_clean_interval [154,162) [v2:127.0.0.1:6810/2086069817,v1:127.0.0.1:6811/2086069817] [v2:127.0.0.1:6812/2086069817,v1:127.0.0.1:6813/2086069817] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:37:26.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:37:26.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:37:26.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:37:26.108 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:37:26.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:37:26.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:37:26.472 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:37:26.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:37:26.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:26.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:37:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767365 2024-03-09T16:37:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767365 2024-03-09T16:37:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365' 2024-03-09T16:37:26.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:26.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:37:26.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=712964571138 2024-03-09T16:37:26.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 712964571138 2024-03-09T16:37:26.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365 1-712964571138' 2024-03-09T16:37:26.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767365 2024-03-09T16:37:26.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:26.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:37:26.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767365 2024-03-09T16:37:26.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:26.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767365 2024-03-09T16:37:26.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-03-09T16:37:26.771 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 687194767365 2024-03-09T16:37:26.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:27.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767365 2024-03-09T16:37:27.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:37:28.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:37:28.045 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:28.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767365 2024-03-09T16:37:28.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:28.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-712964571138 2024-03-09T16:37:28.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:28.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:37:28.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-712964571138 2024-03-09T16:37:28.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:28.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=712964571138 2024-03-09T16:37:28.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 712964571138' 2024-03-09T16:37:28.356 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 712964571138 2024-03-09T16:37:28.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:28.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 712964571138 -lt 712964571138 2024-03-09T16:37:28.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:28.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:28.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:28.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:28.966 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:29.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:29.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:29.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:29.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:29.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:37:29.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:29.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:29.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:29.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:29.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:29.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:29.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:29.668 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:29.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:29.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:29.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:29.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:37:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:37:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:37:30.274 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:37:30.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:37:30.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:37:30.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:37:30.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:37:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:37:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:37:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:37:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:30.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:37:30.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:30.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:30.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:31.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:31.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:31.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:31.105 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:37:31.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:37:31.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:37:31.444 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:37:31.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:37:31.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:37:31.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:37:31.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:31.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:37:32.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:32.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:32.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:32.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:32.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:32.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:32.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:32.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:32.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:37:32.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:37:32.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:37:32.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:37:32.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:37:32.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:37:32.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:37:32.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:37:33.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:37:33.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:37:33.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-09T16:37:33.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:33.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-09T16:37:33.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:33.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:33.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:33.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:33.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:33.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:33.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:33.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:34.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:34.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:34.208 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:34.209 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:34.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:37:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-09T16:37:34.543 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:37:34.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:34.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:34.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:34.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:37:34.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:35.961 INFO:tasks.workunit.client.0.smithi146.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:36.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:36.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:36.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:37:36.820 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:36.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:36.846 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:36.845+0000 7ff632306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:36.848 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:36.847+0000 7ff632306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:36.850 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:36.848+0000 7ff632306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:37.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:37.366 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:37.365+0000 7ff632306640 -1 Falling back to public interface 2024-03-09T16:37:37.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:38.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:38.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:38.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:38.375 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:38.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:38.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:38.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:39.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:39.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:39.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:37:39.658 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:37:39.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:39.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:39.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:39.686+0000 7ff632306640 -1 osd.0 168 log_to_monitors true 2024-03-09T16:37:40.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:37:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:37:41.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:41.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:37:41.316 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 172 up_thru 172 down_at 169 last_clean_interval [160,168) [v2:127.0.0.1:6802/2048384745,v1:127.0.0.1:6803/2048384745] [v2:127.0.0.1:6804/2048384745,v1:127.0.0.1:6805/2048384745] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:37:41.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:37:41.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:37:41.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:37:41.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:37:41.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:37:41.706 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:37:41.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:37:41.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:41.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:37:41.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=738734374914 2024-03-09T16:37:41.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 738734374914 2024-03-09T16:37:41.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374914' 2024-03-09T16:37:41.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:41.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:37:41.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=712964571142 2024-03-09T16:37:41.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 712964571142 2024-03-09T16:37:41.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374914 1-712964571142' 2024-03-09T16:37:41.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:41.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-738734374914 2024-03-09T16:37:41.915 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:41.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:37:41.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-738734374914 2024-03-09T16:37:41.916 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:41.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=738734374914 2024-03-09T16:37:41.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 738734374914' 2024-03-09T16:37:41.917 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 738734374914 2024-03-09T16:37:41.917 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:42.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 738734374914 2024-03-09T16:37:42.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:37:43.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:37:43.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:43.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374914 -lt 738734374914 2024-03-09T16:37:43.526 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:43.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-712964571142 2024-03-09T16:37:43.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:43.527 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:37:43.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-712964571142 2024-03-09T16:37:43.528 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:43.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=712964571142 2024-03-09T16:37:43.529 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 712964571142' 2024-03-09T16:37:43.529 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 712964571142 2024-03-09T16:37:43.530 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:43.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 712964571142 -lt 712964571142 2024-03-09T16:37:43.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:43.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:43.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:44.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:44.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:44.139 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:44.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:44.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:44.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:44.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:44.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:44.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:44.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:44.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:44.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:44.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:44.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:44.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:44.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:37:44.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:44.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:37:45.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:45.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:45.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T16:37:45.854 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:37:45.855 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:37:46.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:37:46.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:46.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:46.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:46.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:46.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:46.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:46.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:46.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:46.435 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:46.434+0000 7f0a2af81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:46.437 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:46.436+0000 7f0a2af81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:46.438 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:46.437+0000 7f0a2af81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:46.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:46.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:46.959 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:46.958+0000 7f0a2af81640 -1 Falling back to public interface 2024-03-09T16:37:46.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:47.965 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:48.250 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:49.032 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:49.030+0000 7f0a2af81640 -1 osd.1 174 log_to_monitors true 2024-03-09T16:37:49.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:49.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:49.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:37:49.252 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:37:49.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:49.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:49.545 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 178 up_thru 172 down_at 175 last_clean_interval [166,174) [v2:127.0.0.1:6810/1935930977,v1:127.0.0.1:6811/1935930977] [v2:127.0.0.1:6812/1935930977,v1:127.0.0.1:6813/1935930977] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:37:49.546 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:37:49.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:37:49.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:37:49.635 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:37:49.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:37:49.946 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:37:49.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:37:49.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:49.947 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:37:50.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=738734374917 2024-03-09T16:37:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 738734374917 2024-03-09T16:37:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374917' 2024-03-09T16:37:50.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:37:50.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:37:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=764504178690 2024-03-09T16:37:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 764504178690 2024-03-09T16:37:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374917 1-764504178690' 2024-03-09T16:37:50.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:50.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-738734374917 2024-03-09T16:37:50.199 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:50.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:37:50.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-738734374917 2024-03-09T16:37:50.200 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:50.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=738734374917 2024-03-09T16:37:50.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 738734374917' 2024-03-09T16:37:50.202 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 738734374917 2024-03-09T16:37:50.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:50.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374915 -lt 738734374917 2024-03-09T16:37:50.530 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:37:51.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:37:51.532 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:37:51.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374917 -lt 738734374917 2024-03-09T16:37:51.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:37:51.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-764504178690 2024-03-09T16:37:51.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:37:51.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:37:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-764504178690 2024-03-09T16:37:51.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:37:51.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=764504178690 2024-03-09T16:37:51.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 764504178690' 2024-03-09T16:37:51.811 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 764504178690 2024-03-09T16:37:51.812 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:37:52.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 764504178690 -lt 764504178690 2024-03-09T16:37:52.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:37:52.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:52.088 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:52.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:37:52.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:52.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:52.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:52.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:52.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:52.887 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:53.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:53.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:37:53.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:37:53.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:37:53.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:53.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:53.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:53.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:53.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:53.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:53.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:53.739 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:37:53.744 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:37:53.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:37:54.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:37:54.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:37:54.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:37:54.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:54.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:37:54.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:54.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:54.275 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:54.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:54.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:54.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:54.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:54.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:54.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:37:54.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:54.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:54.545 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:37:54.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:37:54.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:37:55.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:37:55.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:37:55.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:37:55.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:37:55.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:37:55.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:37:55.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:37:55.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:37:55.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:37:55.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:37:55.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:37:55.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:37:56.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:37:56.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:37:56.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:37:56.271 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:37:56.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:37:56.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:37:56.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:56.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-03-09T16:37:57.330 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:37:57.331 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:37:57.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:37:57.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:37:57.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:37:57.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:37:57.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:37:57.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:37:57.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:37:57.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:37:57.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:37:57.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:37:57.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:37:57.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:37:57.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:37:57.857 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:37:57.857 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:37:57.857 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:37:57.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:57.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:37:57.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:37:57.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:37:57.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:37:57.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:37:57.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:37:57.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:37:57.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:37:57.878 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:57.877+0000 7f50b12b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:57.880 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:57.879+0000 7f50b12b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:57.881 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:57.880+0000 7f50b12b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:37:58.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:37:58.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:58.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:58.400 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:37:58.398+0000 7f50b12b5640 -1 Falling back to public interface 2024-03-09T16:37:58.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:37:59.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:37:59.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:37:59.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:37:59.427 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:37:59.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:37:59.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:37:59.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:00.554 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:00.551+0000 7f50b12b5640 -1 osd.1 180 log_to_monitors true 2024-03-09T16:38:00.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:00.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:00.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:38:00.703 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:38:00.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:00.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:00.991 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 184 up_thru 178 down_at 181 last_clean_interval [178,180) [v2:127.0.0.1:6810/1753449248,v1:127.0.0.1:6811/1753449248] [v2:127.0.0.1:6812/1753449248,v1:127.0.0.1:6813/1753449248] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:38:00.991 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:38:00.992 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:38:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:38:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:38:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:38:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:38:01.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:38:01.080 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:38:01.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:38:01.422 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:38:01.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:38:01.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:01.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:38:01.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=738734374920 2024-03-09T16:38:01.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 738734374920 2024-03-09T16:38:01.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374920' 2024-03-09T16:38:01.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:01.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:38:01.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=790273982466 2024-03-09T16:38:01.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 790273982466 2024-03-09T16:38:01.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-738734374920 1-790273982466' 2024-03-09T16:38:01.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:01.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-738734374920 2024-03-09T16:38:01.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:01.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:38:01.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-738734374920 2024-03-09T16:38:01.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:01.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=738734374920 2024-03-09T16:38:01.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 738734374920' 2024-03-09T16:38:01.674 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 738734374920 2024-03-09T16:38:01.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:01.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374919 -lt 738734374920 2024-03-09T16:38:01.944 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:02.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:38:02.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:03.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374919 -lt 738734374920 2024-03-09T16:38:03.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:04.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:38:04.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:04.501 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 738734374920 -lt 738734374920 2024-03-09T16:38:04.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:04.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-790273982466 2024-03-09T16:38:04.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:04.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:38:04.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-790273982466 2024-03-09T16:38:04.503 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:04.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=790273982466 2024-03-09T16:38:04.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 790273982466' 2024-03-09T16:38:04.505 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 790273982466 2024-03-09T16:38:04.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:04.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 790273982466 -lt 790273982466 2024-03-09T16:38:04.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:38:04.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:04.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:05.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:05.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:05.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:38:05.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:05.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:05.385 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:05.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:38:05.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:38:05.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:38:05.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T16:38:05.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-09T16:38:05.722 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T16:38:05.723 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:38:05.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-09T16:38:05.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-03-09T16:38:06.091 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: {} 2024-03-09T16:38:06.091 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-09T16:38:06.108 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: {} 2024-03-09T16:38:06.109 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:06.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-09T16:38:06.120 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:06.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:06.408 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:06.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T16:38:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:06.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:06.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:06.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:06.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-09T16:38:06.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-09T16:38:06.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:06.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-09T16:38:06.590 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:06.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:06.860 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:06.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:06.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T16:38:06.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:06.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:06.863 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:06.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:06.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:06.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:06.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:07.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T16:38:07.041 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:07.319 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:07.320 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:07.320 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:07.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:07.320 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:07.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T16:38:07.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:07.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T16:38:07.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:07.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:07.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:07.323 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T16:38:07.323 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:07.323 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:07.324 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:07.324 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:07.324 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:07.324 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:07.325 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:07.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:07.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:07.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T16:38:07.494 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:07.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T16:38:07.773 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:07.774 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:07.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:07.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:07.861 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:07.862 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:07.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:07.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:07.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:07.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:08.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:08.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T16:38:08.472 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-09T16:38:08.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:08.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:08.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:08.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:08.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:09.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:09.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:10.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:10.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:11.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:11.031 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:11.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:08.578349+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:11.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:38:11.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-03-09T16:38:11.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-03-09T16:38:11.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-09T16:38:11.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-09T16:38:11.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T16:38:11.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-03-09T16:38:11.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-03-09T16:38:11.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-03-09T16:38:11.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.397 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.443 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:11.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:11.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-09T16:38:11.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:38:11.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-09T16:38:11.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:38:11.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-09T16:38:11.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-09T16:38:11.571 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=184 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:38:11.574 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:38:11.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T16:38:11.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:38:11.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:38:11.598 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:38:11.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T16:38:11.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-03-09T16:38:11.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-09T16:38:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:38:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:38:11.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:38:11.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:38:11.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:38:11.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:38:11.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:38:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:38:11.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:38:11.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:38:11.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:38:11.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:38:11.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:38:11.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:38:11.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:38:11.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T16:38:12.638 INFO:tasks.workunit.client.0.smithi146.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:38:13.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:38:13.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:13.158 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:38:13.159 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:38:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:38:13.160 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:38:13.160 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:38:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:38:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:38:13.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:38:13.186 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:13.183+0000 7f029c3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:13.195 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:13.187+0000 7f029c3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:13.195 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:13.189+0000 7f029c3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:13.440 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:13.441 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:38:13.442 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:38:13.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:13.442 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:13.721 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:13.719+0000 7f029c3ac640 -1 Falling back to public interface 2024-03-09T16:38:13.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:14.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:15.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:15.812 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:15.811+0000 7f029c3ac640 -1 osd.0 185 log_to_monitors true 2024-03-09T16:38:16.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:16.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:16.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:38:16.057 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:38:16.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:16.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 189 up_thru 184 down_at 186 last_clean_interval [172,185) [v2:127.0.0.1:6802/529410056,v1:127.0.0.1:6803/529410056] [v2:127.0.0.1:6804/529410056,v1:127.0.0.1:6805/529410056] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:38:16.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:38:16.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:38:16.547 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:38:16.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:38:16.905 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:38:16.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:38:16.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:16.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:38:17.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-03-09T16:38:17.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-03-09T16:38:17.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-03-09T16:38:17.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:17.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:38:17.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=790273982470 2024-03-09T16:38:17.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 790273982470 2024-03-09T16:38:17.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-790273982470' 2024-03-09T16:38:17.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:17.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-03-09T16:38:17.132 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:17.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:38:17.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-03-09T16:38:17.133 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:17.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-03-09T16:38:17.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-03-09T16:38:17.136 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 811748818946 2024-03-09T16:38:17.136 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:17.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-03-09T16:38:17.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:17.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-790273982470 2024-03-09T16:38:17.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:17.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:38:17.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-790273982470 2024-03-09T16:38:17.408 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:17.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=790273982470 2024-03-09T16:38:17.409 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 790273982470' 2024-03-09T16:38:17.409 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 790273982470 2024-03-09T16:38:17.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:17.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 790273982469 -lt 790273982470 2024-03-09T16:38:17.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:18.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:38:18.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:18.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 790273982469 -lt 790273982470 2024-03-09T16:38:18.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:19.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:38:19.970 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:20.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 790273982470 -lt 790273982470 2024-03-09T16:38:20.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:38:20.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:20.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:20.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:20.595 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:20.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:38:20.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:20.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:20.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:21.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:38:21.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:38:21.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:38:21.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-09T16:38:21.201 INFO:tasks.workunit.client.0.smithi146.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-09T16:38:21.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T16:38:21.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T16:38:21.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:38:21.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:38:21.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:38:21.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:38:21.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:38:21.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:38:21.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:38:21.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T16:38:21.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:38:21.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:38:21.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:38:21.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:38:21.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:38:21.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-03-09T16:38:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:38:22.260 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:38:22.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:38:22.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:38:22.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:38:22.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:38:22.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:38:22.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:38:22.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:38:22.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:38:22.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:38:22.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:38:22.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:38:22.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:38:22.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:38:22.565 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:22.563+0000 7fd6b0b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:22.567 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:22.565+0000 7fd6b0b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:22.568 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:22.567+0000 7fd6b0b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:38:22.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:22.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:23.082 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:23.081+0000 7fd6b0b8c640 -1 Falling back to public interface 2024-03-09T16:38:23.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:24.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:24.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:24.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:38:24.094 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:38:24.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:24.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:24.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:25.170 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:25.168+0000 7fd6b0b8c640 -1 osd.1 190 log_to_monitors true 2024-03-09T16:38:25.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:25.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:25.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:38:25.370 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:38:25.370 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:25.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:25.672 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 194 up_thru 189 down_at 191 last_clean_interval [184,190) [v2:127.0.0.1:6810/1721920347,v1:127.0.0.1:6811/1721920347] [v2:127.0.0.1:6812/1721920347,v1:127.0.0.1:6813/1721920347] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:38:25.673 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:38:25.674 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:38:25.674 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:38:25.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:38:25.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:38:25.674 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:38:25.762 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:38:26.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:38:26.036 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:38:26.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:38:26.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:26.037 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:38:26.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818949 2024-03-09T16:38:26.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818949 2024-03-09T16:38:26.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949' 2024-03-09T16:38:26.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:26.142 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:38:26.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655426 2024-03-09T16:38:26.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655426 2024-03-09T16:38:26.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949 1-833223655426' 2024-03-09T16:38:26.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:26.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818949 2024-03-09T16:38:26.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:26.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:38:26.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818949 2024-03-09T16:38:26.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:26.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818949 2024-03-09T16:38:26.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818949' 2024-03-09T16:38:26.268 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 811748818949 2024-03-09T16:38:26.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:26.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818947 -lt 811748818949 2024-03-09T16:38:26.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:27.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:38:27.568 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:27.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818949 -lt 811748818949 2024-03-09T16:38:27.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:27.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-833223655426 2024-03-09T16:38:27.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:27.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:38:27.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-833223655426 2024-03-09T16:38:27.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655426 2024-03-09T16:38:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 833223655426' 2024-03-09T16:38:27.844 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 833223655426 2024-03-09T16:38:27.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:28.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655426 -lt 833223655426 2024-03-09T16:38:28.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:38:28.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:28.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:28.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:38:28.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:28.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:28.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:28.495 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:28.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:28.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:28.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:28.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:38:28.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:28.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:28.764 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:29.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:38:29.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T16:38:29.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:38:29.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T16:38:29.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:38:29.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:38:29.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:38:29.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:29.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:29.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:29.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:29.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:29.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:29.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:29.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:38:29.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:29.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:29.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:29.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:38:29.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:38:30.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:38:30.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:38:30.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:38:30.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:38:30.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:38:30.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:38:30.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:30.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:30.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T16:38:30.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:30.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:30.610 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:30.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:38:30.948 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:38:31.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:38:31.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:38:31.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:38:31.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:38:31.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:31.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:31.702 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:31.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:38:31.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:31.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:31.971 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:32.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:38:32.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:38:32.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:38:32.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-03-09T16:38:32.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:38:32.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:38:32.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:38:32.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:38:32.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:38:32.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:33.566 INFO:tasks.workunit.client.0.smithi146.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T16:38:34.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:38:34.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:38:34.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:38:34.085 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:38:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:38:34.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:38:34.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:38:34.087 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:38:34.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:38:34.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:38:34.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:38:34.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:38:34.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:38:34.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:38:34.112 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:34.111+0000 7fd65fa1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:34.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:34.113+0000 7fd65fa1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:34.116 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:34.114+0000 7fd65fa1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:34.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:38:34.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:34.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:34.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:34.635 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:34.634+0000 7fd65fa1c640 -1 Falling back to public interface 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:35.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:35.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:36.798 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:36.797+0000 7fd65fa1c640 -1 osd.0 196 log_to_monitors true 2024-03-09T16:38:36.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:38:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:38:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:36.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 200 up_thru 195 down_at 197 last_clean_interval [189,196) [v2:127.0.0.1:6802/1547106327,v1:127.0.0.1:6803/1547106327] [v2:127.0.0.1:6804/1547106327,v1:127.0.0.1:6805/1547106327] exists,up f819a490-4e68-47ab-858e-7bf51516a271 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:38:37.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:38:37.190 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:38:37.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:38:37.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:38:37.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:38:37.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:38:37.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:38:37.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:38:37.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:38:37.628 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:38:37.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:38:37.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:37.628 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:38:37.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459202 2024-03-09T16:38:37.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459202 2024-03-09T16:38:37.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459202' 2024-03-09T16:38:37.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:37.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655429 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655429 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459202 1-833223655429' 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459202 2024-03-09T16:38:37.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:37.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:38:37.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459202 2024-03-09T16:38:37.939 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459202 2024-03-09T16:38:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459202' 2024-03-09T16:38:37.940 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 858993459202 2024-03-09T16:38:37.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:38.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 858993459202 2024-03-09T16:38:38.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:39.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:38:39.256 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:39.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459202 -lt 858993459202 2024-03-09T16:38:39.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:39.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-833223655429 2024-03-09T16:38:39.541 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:39.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:38:39.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-833223655429 2024-03-09T16:38:39.542 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:39.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655429 2024-03-09T16:38:39.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 833223655429' 2024-03-09T16:38:39.544 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 833223655429 2024-03-09T16:38:39.544 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:39.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655429 -lt 833223655429 2024-03-09T16:38:39.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:38:39.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:39.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:40.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:40.451 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:38:40.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:40.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:40.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:38:40.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:38:40.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:38:40.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:38:41.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:38:41.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T16:38:41.936 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T16:38:41.936 INFO:tasks.workunit.client.0.smithi146.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T16:38:42.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:38:42.453 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:38:42.454 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:38:42.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:38:42.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:38:42.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:38:42.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:38:42.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:38:42.485 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:42.483+0000 7f3fdb00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:42.487 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:42.486+0000 7f3fdb00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:42.488 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:42.487+0000 7f3fdb00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:38:42.737 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:38:42.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:42.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:43.002 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:43.001+0000 7f3fdb00d640 -1 Falling back to public interface 2024-03-09T16:38:43.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:44.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:44.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:38:45.103 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:38:45.102+0000 7f3fdb00d640 -1 osd.1 202 log_to_monitors true 2024-03-09T16:38:45.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:38:45.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:38:45.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:38:45.286 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:38:45.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:38:45.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:38:45.716 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 206 up_thru 201 down_at 203 last_clean_interval [194,202) [v2:127.0.0.1:6810/3445537303,v1:127.0.0.1:6811/3445537303] [v2:127.0.0.1:6812/3445537303,v1:127.0.0.1:6813/3445537303] exists,up 214f63d9-be29-4ef6-bc2e-2c1555c1e4b9 2024-03-09T16:38:45.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:38:45.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:38:45.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:38:45.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:38:45.717 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:38:45.718 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:38:45.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:38:46.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:38:46.170 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:38:46.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:38:46.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:46.170 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:38:46.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459204 2024-03-09T16:38:46.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459204 2024-03-09T16:38:46.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459204' 2024-03-09T16:38:46.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:38:46.273 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:38:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262978 2024-03-09T16:38:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262978 2024-03-09T16:38:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459204 1-884763262978' 2024-03-09T16:38:46.378 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:46.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459204 2024-03-09T16:38:46.378 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:46.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:38:46.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459204 2024-03-09T16:38:46.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:46.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459204 2024-03-09T16:38:46.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459204' 2024-03-09T16:38:46.381 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 858993459204 2024-03-09T16:38:46.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:46.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459203 -lt 858993459204 2024-03-09T16:38:46.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:38:47.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:38:47.654 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:38:47.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459205 -lt 858993459204 2024-03-09T16:38:47.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:38:47.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-884763262978 2024-03-09T16:38:47.938 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:38:47.939 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:38:47.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-884763262978 2024-03-09T16:38:47.940 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:38:47.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262978 2024-03-09T16:38:47.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 884763262978' 2024-03-09T16:38:47.941 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 884763262978 2024-03-09T16:38:47.942 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:38:48.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262978 -lt 884763262978 2024-03-09T16:38:48.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:38:48.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:48.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:38:48.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:38:48.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:38:48.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:38:48.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:38:48.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:38:49.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:38:49.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:38:49.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:38:49.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-03-09T16:38:49.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-09T16:38:49.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:49.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:49.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-09T16:38:49.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-09T16:38:49.190 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:49.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:49.465 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:49.465 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:49.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:49.465 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:49.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T16:38:49.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:49.466 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:49.466 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:49.467 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T16:38:49.468 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:49.468 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:49.468 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:49.468 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:49.468 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:49.469 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:49.469 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:49.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:49.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:49.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-09T16:38:49.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-09T16:38:49.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:49.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-09T16:38:49.646 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-09T16:38:49.646 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:49.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:49.920 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:49.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:49.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:49.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:49.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T16:38:49.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:49.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:49.921 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:49.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:49.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T16:38:49.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:49.923 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:49.926 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:49.926 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:49.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:50.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:50.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:50.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:50.012 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-09T16:38:50.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:50.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T16:38:50.102 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T16:38:50.102 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:50.398 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:50.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:50.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:50.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:50.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T16:38:50.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:50.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:50.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:50.401 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:50.402 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:50.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:50.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:50.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:50.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T16:38:50.490 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.0.asok 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:50.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T16:38:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T16:38:50.579 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T16:38:50.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T16:38:50.580 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T16:38:50.580 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T16:38:50.580 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:38:50.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T16:38:50.852 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:38:50.852 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T16:38:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T16:38:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T16:38:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T16:38:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T16:38:50.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T16:38:50.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T16:38:50.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T16:38:50.855 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T16:38:50.855 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:50.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69596/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T16:38:50.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T16:38:50.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-osd.1.asok 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T16:38:50.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69596/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-09T16:38:51.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-03-09T16:38:51.128 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0: {} 2024-03-09T16:38:51.128 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0: osd_max_object_size = '' 2024-03-09T16:38:51.145 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1: {} 2024-03-09T16:38:51.146 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1: osd_max_object_size = '' 2024-03-09T16:38:51.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-09T16:38:51.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T16:38:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:51.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:51.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:51.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:51.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T16:38:51.722 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-09T16:38:51.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T16:34:16.056819+0000 last_deep_scrub_stamp 2024-03-09T16:38:51.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:38:51.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:51.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:51.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:51.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:52.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:52.000 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:53.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:53.001 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:53.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:53.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:53.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:53.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:53.002 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:53.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:53.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:54.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:54.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:54.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:54.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:54.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:54.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:54.285 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:54.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:54.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:55.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:55.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:55.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:55.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:55.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:55.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:55.566 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:55.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:55.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:56.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:56.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:56.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:56.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:56.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:56.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:56.844 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:57.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:34:16.056819+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:57.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:38:58.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:38:58.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:58.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T16:38:58.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:58.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:38:58.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:58.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:34:16.056819+0000 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-03-09T16:38:58.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-03-09T16:38:58.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-09T16:38:58.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-03-09T16:38:58.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.477 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.600 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.702 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.731 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T16:38:58.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T16:38:58.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-09T16:38:58.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:38:58.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-09T16:38:58.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:38:58.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-09T16:38:58.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-09T16:38:58.868 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-09T16:38:58.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=206 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:38:58.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T16:38:58.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:38:58.896 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T16:38:58.896 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:38:58.896 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:38:58.896 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:38:58.896 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T16:38:58.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:38:58.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:38:58.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:38:58.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-03-09T16:38:58.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-09T16:38:58.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-03-09T16:38:58.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-03-09T16:38:58.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-03-09T16:38:58.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:58.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:58.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:58.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:59.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:38:52.384369+0000 2024-03-09T16:38:59.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-03-09T16:38:59.444 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0 on osd.1 to repair 2024-03-09T16:38:59.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-03-09T16:38:52.384369+0000 2024-03-09T16:38:59.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:38:59.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:38:52.384369+0000 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:38:59.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:38:59.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:38:59.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:00.725 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:00.726 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:01.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:01.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:02.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:02.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:02.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:02.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:02.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:02.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:02.377 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:02.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:02.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:03.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:03.651 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:03.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:03.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:03.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:03.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:03.652 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:03.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:03.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:04.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:04.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:04.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:04.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:04.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:04.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:04.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:05.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:05.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:06.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:06.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:06.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:06.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:06.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:06.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:06.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:06.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:38:52.384369+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:06.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:39:07.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:39:07.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:39:01.823254+0000 '>' 2024-03-09T16:38:52.384369+0000 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:39:07.767 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:39:07.768 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:39:07.768 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:39:07.768 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:39:07.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:39:07.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:39:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:39:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:39:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:39:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:39:07.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:39:07.857 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:39:08.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:39:08.127 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:39:08.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:39:08.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:39:08.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:39:08.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459210 2024-03-09T16:39:08.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459210 2024-03-09T16:39:08.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459210' 2024-03-09T16:39:08.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:39:08.270 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262983 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262983 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-858993459210 1-884763262983' 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-858993459210 2024-03-09T16:39:08.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:39:08.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:39:08.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-858993459210 2024-03-09T16:39:08.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:39:08.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459210 2024-03-09T16:39:08.376 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 858993459210 2024-03-09T16:39:08.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 858993459210' 2024-03-09T16:39:08.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:39:08.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459209 -lt 858993459210 2024-03-09T16:39:08.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:39:09.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:39:09.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:39:09.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459210 -lt 858993459210 2024-03-09T16:39:09.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:39:09.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-884763262983 2024-03-09T16:39:09.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:39:09.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:39:09.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-884763262983 2024-03-09T16:39:09.925 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:39:09.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262983 2024-03-09T16:39:09.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 884763262983' 2024-03-09T16:39:09.926 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 884763262983 2024-03-09T16:39:09.926 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:39:10.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262983 -lt 884763262983 2024-03-09T16:39:10.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:39:10.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:39:10.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:39:10.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:39:10.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:39:10.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:39:10.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:39:10.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:39:11.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:39:11.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:39:11.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:39:11.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T16:39:11.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T16:39:11.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T16:39:11.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T16:39:11.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T16:39:11.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T16:39:11.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-03-09T16:39:11.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-09T16:39:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'csr_pool' removed 2024-03-09T16:39:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:39:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:39:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:39:11.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:39:11.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:39:11.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:39:11.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:39:11.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:39:11.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:39:11.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:39:11.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:39:11.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:39:11.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:39:11.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:39:11.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:39:11.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:39:11.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:39:11.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:39:11.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:39:11.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:39:11.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:39:11.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:39:11.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:39:11.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:39:11.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:11.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:11.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:39:11.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:39:11.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:39:11.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:39:11.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:39:11.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:39:11.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:39:11.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:39:11.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:39:11.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:39:11.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:39:11.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:39:11.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:39:11.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:39:11.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:39:11.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:39:11.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:39:11.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:39:11.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:39:11.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:39:11.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:39:11.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:39:11.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:11.851 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:11.851 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:39:11.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:39:11.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:39:11.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:39:11.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:39:11.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:11.853 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-03-09T16:39:11.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:39:11.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T16:39:12.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:39:12.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:39:12.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:39:12.237 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:39:12.237 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:12.237 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:12.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:12.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T16:39:12.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:39:12.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:39:12.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:39:12.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:39:12.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:39:12.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:39:12.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:39:12.512 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:39:12.513 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:39:12.513 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:12.513 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:12.513 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:39:12.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:39:12.514 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:39:12.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-03-09T16:39:12.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:39:12.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:39:12.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:39:12.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:39:12.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:39:12.598 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:13.397 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:13.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:39:13.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:39:13.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:39:13.428 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:39:13.429 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:39:13.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e0f4100e-8230-4115-9ce3-1001650759b3 2024-03-09T16:39:13.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e0f4100e-8230-4115-9ce3-1001650759b3' 2024-03-09T16:39:13.430 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 e0f4100e-8230-4115-9ce3-1001650759b3 2024-03-09T16:39:13.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:39:13.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxkOxlnk17GhAAVJRQFqZqpPJJyHtVkv24XA== 2024-03-09T16:39:13.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxkOxlnk17GhAAVJRQFqZqpPJJyHtVkv24XA=="}' 2024-03-09T16:39:13.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e0f4100e-8230-4115-9ce3-1001650759b3 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:39:13.652 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:39:13.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:39:13.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxkOxlnk17GhAAVJRQFqZqpPJJyHtVkv24XA== --osd-uuid e0f4100e-8230-4115-9ce3-1001650759b3 2024-03-09T16:39:13.691 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.689+0000 7ff781019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:13.703 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.700+0000 7ff781019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.701+0000 7ff781019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.702+0000 7ff781019640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:39:13.704 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.702+0000 7ff781019640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:39:13.705 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.702+0000 7ff781019640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:39:13.787 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:13.784+0000 7ff781019640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:39:19.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:39:19.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:39:19.368 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:39:19.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:39:19.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:39:19.732 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:39:19.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:39:19.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:39:19.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:39:19.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:39:19.733 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:39:19.758 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:19.756+0000 7fb0b0a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:19.758+0000 7fb0b0a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:19.762 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:19.759+0000 7fb0b0a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:20.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:39:20.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:39:20.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:39:20.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:39:20.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:39:20.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:20.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:39:20.009 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:39:20.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:20.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:20.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:20.291 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:20.289+0000 7fb0b0a11640 -1 Falling back to public interface 2024-03-09T16:39:21.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:21.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:21.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:39:21.278 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:39:21.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:21.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:21.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:22.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:22.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:22.556 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:39:22.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:39:22.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:22.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:22.642 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:22.640+0000 7fb0b0a11640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:39:22.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:23.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:23.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:23.902 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:39:23.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:39:23.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:23.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:24.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:25.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:25.190 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:39:25.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:25.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:39:25.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:25.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:25.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:26.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:26.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:26.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:39:26.591 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:39:26.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:26.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:26.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:27.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:27.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:27.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:39:27.869 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:39:27.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:27.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:28.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:29.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:29.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:29.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:39:29.219 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:39:29.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:29.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:29.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:30.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:30.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:30.549 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:39:30.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:39:30.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:30.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:30.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:31.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:31.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:31.833 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:39:31.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:39:31.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:31.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:32.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:33.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:33.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:33.115 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:39:33.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:39:33.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:33.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:33.446 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:34.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:34.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:34.448 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:39:34.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:39:34.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:34.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:35.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:36.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:36.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:36.043 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:39:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:39:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:36.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:37.794 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:39:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:39:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:38.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:39.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:39.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:39.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:39:39.401 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:39:39.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:39.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:40.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:41.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:41.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:41.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:39:41.264 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:39:41.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:41.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:41.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:42.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:42.729 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:42.730 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:39:42.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:39:42.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:42.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:43.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:44.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:44.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:44.410 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:39:44.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:39:44.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:44.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:44.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:45.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:45.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:45.696 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:39:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:39:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:45.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:39:45.988 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/49573200,v1:127.0.0.1:6803/49573200] [v2:127.0.0.1:6804/49573200,v1:127.0.0.1:6805/49573200] exists,up e0f4100e-8230-4115-9ce3-1001650759b3 2024-03-09T16:39:45.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:39:45.993 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:39:45.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43a74b07-09bc-47f7-89d8-5498720b48aa 2024-03-09T16:39:45.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 43a74b07-09bc-47f7-89d8-5498720b48aa' 2024-03-09T16:39:45.996 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 43a74b07-09bc-47f7-89d8-5498720b48aa 2024-03-09T16:39:45.996 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:39:46.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSkOxlazKKABAAK6rM5/kNTqygB24QQGyTSQ== 2024-03-09T16:39:46.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSkOxlazKKABAAK6rM5/kNTqygB24QQGyTSQ=="}' 2024-03-09T16:39:46.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43a74b07-09bc-47f7-89d8-5498720b48aa -i td/osd-scrub-repair/1/new.json 2024-03-09T16:39:46.328 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:39:46.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:39:46.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSkOxlazKKABAAK6rM5/kNTqygB24QQGyTSQ== --osd-uuid 43a74b07-09bc-47f7-89d8-5498720b48aa 2024-03-09T16:39:46.370 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.367+0000 7fa64b012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.403+0000 7fa64b012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:46.406 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.404+0000 7fa64b012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.405+0000 7fa64b012640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:39:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.405+0000 7fa64b012640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:39:46.407 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.405+0000 7fa64b012640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:39:46.471 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:46.469+0000 7fa64b012640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:39:52.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:39:52.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:39:52.468 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:39:52.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:39:52.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:39:52.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:39:52.843 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:39:52.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:39:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:39:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:39:52.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:39:52.869 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:52.866+0000 7f13a3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:52.870 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:52.868+0000 7f13a3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:52.871 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:52.869+0000 7f13a3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:39:53.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:39:53.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:39:53.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:39:53.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:39:53.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:39:53.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:53.120 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:39:53.121 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:39:53.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:53.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:53.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:53.388 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:53.387+0000 7f13a3c14640 -1 Falling back to public interface 2024-03-09T16:39:54.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:54.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:54.388 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:39:54.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:39:54.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:54.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:54.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:55.669 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:55.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:55.670 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:39:55.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:39:55.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:55.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:55.775 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:39:55.773+0000 7f13a3c14640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:39:56.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:57.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:57.018 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:57.018 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:39:57.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:39:57.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:57.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:57.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:58.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:58.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:58.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:39:58.299 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:39:58.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:58.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:58.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:39:59.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:39:59.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:39:59.630 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:39:59.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:39:59.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:39:59.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:39:59.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:00.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:00.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:00.914 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:40:00.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:40:00.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:00.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:01.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:02.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:02.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:02.197 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:40:02.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:40:02.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:02.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:02.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:03.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:03.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:03.478 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:40:03.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:40:03.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:03.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:03.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:04.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:04.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:04.774 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:40:04.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:40:04.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:04.774 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:05.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:06.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:06.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:06.058 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:40:06.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:40:06.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:06.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:06.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:07.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:07.342 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:40:07.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:40:07.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:07.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:07.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:08.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:08.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:08.623 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:40:08.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:40:08.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:08.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:09.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:10.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:10.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:10.305 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:40:10.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:40:10.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:10.305 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:10.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:11.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:11.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:11.780 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:40:11.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:40:11.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:11.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:12.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:13.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:13.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:13.484 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:40:13.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:40:13.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:13.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:14.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:15.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:15.227 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:15.228 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:40:15.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:40:15.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:15.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:15.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:16.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:16.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:16.848 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:40:16.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:40:16.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:16.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:17.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:18.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:18.709 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:40:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:40:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:18.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:19.002 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4099930627,v1:127.0.0.1:6811/4099930627] [v2:127.0.0.1:6812/4099930627,v1:127.0.0.1:6813/4099930627] exists,up 43a74b07-09bc-47f7-89d8-5498720b48aa 2024-03-09T16:40:19.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:40:19.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:40:19.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:40:19.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-03-09T16:40:19.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:40:19.288 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:40:19.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:40:19.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:40:19.784 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:40:19.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:40:20.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:40:22.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-09T16:40:22.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:40:22.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:40:22.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:40:22.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:40:22.351 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:40:22.351 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:40:22.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:40:22.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:40:22.351 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:40:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:40:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:40:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:40:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:40:22.452 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:40:22.453 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:40:22.723 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:40:22.723 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:40:22.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:40:22.724 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:22.724 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:40:22.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-09T16:40:22.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-09T16:40:22.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-09T16:40:22.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:22.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:40:22.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-09T16:40:22.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-09T16:40:22.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672968' 2024-03-09T16:40:22.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:22.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-09T16:40:22.933 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:22.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:40:22.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-09T16:40:22.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:22.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-09T16:40:22.936 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836491 2024-03-09T16:40:22.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-09T16:40:22.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:23.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-03-09T16:40:23.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:40:24.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:40:24.206 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:24.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-09T16:40:24.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:24.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-09T16:40:24.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:24.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:40:24.489 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-09T16:40:24.489 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:24.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-09T16:40:24.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-09T16:40:24.491 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672968 2024-03-09T16:40:24.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:24.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-09T16:40:24.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:40:24.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:24.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:40:25.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:40:25.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:40:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:40:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:40:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:40:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:40:25.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:40:25.393 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:40:25.393 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:40:25.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:25.394 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:40:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:40:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:40:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-03-09T16:40:25.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T16:40:26.118 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'foo' created 2024-03-09T16:40:26.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:40:27.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-03-09T16:40:27.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-09T16:40:27.766 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'csr_pool' created 2024-03-09T16:40:27.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:40:28.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-09T16:40:28.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:40:28.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:40:28.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:40:28.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:40:28.781 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:40:28.781 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:40:28.781 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:40:28.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:40:28.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:40:28.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:40:28.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:40:28.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:40:28.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:40:28.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:40:28.883 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:40:29.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:40:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:40:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:40:29.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:29.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:40:29.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-09T16:40:29.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-09T16:40:29.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-09T16:40:29.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:29.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:40:29.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-09T16:40:29.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-09T16:40:29.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2024-03-09T16:40:29.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:29.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-09T16:40:29.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:29.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:40:29.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-09T16:40:29.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:29.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-09T16:40:29.463 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836493 2024-03-09T16:40:29.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-09T16:40:29.463 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:29.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-09T16:40:29.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:29.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-09T16:40:29.735 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:29.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:40:29.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-09T16:40:29.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:29.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-09T16:40:29.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-09T16:40:29.738 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672971 2024-03-09T16:40:29.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:30.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672971 2024-03-09T16:40:30.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:40:31.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:40:31.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:31.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672971 2024-03-09T16:40:31.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:40:32.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:40:32.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:32.597 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-09T16:40:32.597 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:40:32.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:32.598 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:32.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:40:32.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:40:33.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:40:33.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:40:33.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:33.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:33.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:40:33.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:40:33.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:40:33.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-09T16:40:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T16:40:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-09T16:40:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-09T16:40:33.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:40:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:40:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-09T16:40:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:40:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:40:33.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:40:33.920 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:40:33.931 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:40:34.296 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:40:34.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:40:34.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:40:34.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:40:34.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-09T16:40:34.743 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-09T16:40:34.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T16:40:34.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-09T16:40:34.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:40:34.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:40:35.208 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:40:35.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:40:35.651 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:40:35.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:40:35.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:40:35.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:40:35.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-09T16:40:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-09T16:40:36.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-03-09T16:40:36.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T16:40:36.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T16:40:36.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T16:40:36.232 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:40:36.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-03-09T16:40:36.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-03-09T16:40:36.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-09T16:40:36.525 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-09T16:40:36.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-09T16:40:36.526 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:40:36.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-03-09T16:40:36.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-03-09T16:40:37.023 INFO:tasks.workunit.client.0.smithi146.stdout:created pool csr_pool snap snap1 2024-03-09T16:40:37.025 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-03-09T16:40:37.026 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-09T16:40:37.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T16:40:37.027 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-09T16:40:37.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-03-09T16:40:37.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-03-09T16:40:37.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-09T16:40:37.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:40:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:40:37.254 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:40:37.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:40:37.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:40:37.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-03-09T16:40:39.559 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:40:39.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:40:39.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:40:39.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:40:39.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:40:39.562 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:40:39.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:40:39.590 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:39.588+0000 7fcaeaa2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:39.592 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:39.591+0000 7fcaeaa2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:39.593 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:39.592+0000 7fcaeaa2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:40:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:40:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:40:39.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:39.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:40.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:40.117 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:40.116+0000 7fcaeaa2a640 -1 Falling back to public interface 2024-03-09T16:40:41.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:41.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:41.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:40:41.113 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:40:41.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:41.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:41.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:42.391 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:42.459 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:42.458+0000 7fcaeaa2a640 -1 osd.1 29 log_to_monitors true 2024-03-09T16:40:42.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:43.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:40:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/1402598466,v1:127.0.0.1:6811/1402598466] [v2:127.0.0.1:6812/1402598466,v1:127.0.0.1:6813/1402598466] exists,up 43a74b07-09bc-47f7-89d8-5498720b48aa 2024-03-09T16:40:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:40:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:40:43.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:40:43.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:40:44.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:40:44.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:40:44.040 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:40:44.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:40:44.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:40:44.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:40:44.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:40:44.436 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:40:44.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:40:44.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:44.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:40:44.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-09T16:40:44.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-09T16:40:44.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-09T16:40:44.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:44.538 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:40:44.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-03-09T16:40:44.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-03-09T16:40:44.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-141733920770' 2024-03-09T16:40:44.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:44.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-09T16:40:44.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:44.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:40:44.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-09T16:40:44.645 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:44.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-09T16:40:44.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-09T16:40:44.646 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 21474836497 2024-03-09T16:40:44.647 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:44.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-09T16:40:44.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:40:45.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:40:45.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:46.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836497 2024-03-09T16:40:46.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:46.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920770 2024-03-09T16:40:46.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:46.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:40:46.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920770 2024-03-09T16:40:46.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:46.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-03-09T16:40:46.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2024-03-09T16:40:46.204 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 141733920770 2024-03-09T16:40:46.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:46.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920770 -lt 141733920770 2024-03-09T16:40:46.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:40:46.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:46.475 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:46.847 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:40:46.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:40:47.116 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:40:47.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:40:47.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:47.116 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-09T16:40:47.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-03-09T16:40:47.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-03-09T16:40:47.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-09T16:40:47.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:40:47.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:40:47.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:40:47.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:40:47.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:40:47.895 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-03-09T16:40:49.753 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:40:49.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:40:49.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:40:49.756 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:40:49.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:40:49.782 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:49.781+0000 7f8399ac2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:49.785 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:49.784+0000 7f8399ac2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:49.786 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:49.785+0000 7f8399ac2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:40:50.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:40:50.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:40:50.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:50.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:40:50.300 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:50.299+0000 7f8399ac2640 -1 Falling back to public interface 2024-03-09T16:40:50.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:51.313 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:40:51.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:52.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:40:52.757 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:40:52.756+0000 7f8399ac2640 -1 osd.0 35 log_to_monitors true 2024-03-09T16:40:52.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:40:53.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:40:53.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:40:53.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:40:53.937 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:40:53.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:40:53.938 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/2354252502,v1:127.0.0.1:6803/2354252502] [v2:127.0.0.1:6804/2354252502,v1:127.0.0.1:6805/2354252502] exists,up e0f4100e-8230-4115-9ce3-1001650759b3 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:40:54.215 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:40:54.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:40:54.216 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:40:54.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:40:54.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:40:54.216 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:40:54.306 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:40:54.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:40:54.592 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:40:54.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:40:54.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:54.592 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:40:54.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-09T16:40:54.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-09T16:40:54.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546' 2024-03-09T16:40:54.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:40:54.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:40:54.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-03-09T16:40:54.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-03-09T16:40:54.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-141733920773' 2024-03-09T16:40:54.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:54.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724546 2024-03-09T16:40:54.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:54.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:40:54.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724546 2024-03-09T16:40:54.801 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:54.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-09T16:40:54.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-03-09T16:40:54.803 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 167503724546 2024-03-09T16:40:54.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:55.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-03-09T16:40:55.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:40:56.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:40:56.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:40:56.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-09T16:40:56.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:40:56.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920773 2024-03-09T16:40:56.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:40:56.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:40:56.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920773 2024-03-09T16:40:56.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:40:56.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-03-09T16:40:56.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2024-03-09T16:40:56.357 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 141733920773 2024-03-09T16:40:56.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:40:56.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920773 -lt 141733920773 2024-03-09T16:40:56.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:40:56.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:56.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:56.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T16:40:56.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:40:56.995 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:40:57.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T16:40:57.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:40:57.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:40:57.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:40:57.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T16:40:57.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:40:57.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:40:57.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-03-09T16:40:57.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-03-09T16:40:57.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T16:40:57.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T16:40:57.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:40:57.630 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:40:57.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:40:57.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:40:57.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T16:40:27.631071+0000 2024-03-09T16:40:57.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T16:40:58.179 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-09T16:40:58.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T16:40:27.631071+0000 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:40:27.631071+0000 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:40:58.191 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:40:58.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:40:27.631071+0000 '>' 2024-03-09T16:40:27.631071+0000 2024-03-09T16:40:58.457 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:40:59.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:40:59.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:40:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:40:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:40:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:40:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:40:59.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:40:59.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:40:27.631071+0000 '>' 2024-03-09T16:40:27.631071+0000 2024-03-09T16:40:59.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:41:00.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:41:00.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:41:00.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:41:00.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:41:00.802 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:41:00.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:41:00.803 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:41:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:40:27.631071+0000 '>' 2024-03-09T16:40:27.631071+0000 2024-03-09T16:41:01.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:41:02.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:41:02.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:41:02.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:41:02.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:41:02.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:41:02.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:41:02.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:41:02.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:40:27.631071+0000 '>' 2024-03-09T16:40:27.631071+0000 2024-03-09T16:41:02.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:41:03.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:41:03.353 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:41:03.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T16:41:03.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T16:41:03.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:41:03.353 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:41:03.354 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T16:41:03.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:40:58.568025+0000 '>' 2024-03-09T16:40:27.631071+0000 2024-03-09T16:41:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:41:03.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-03-09T16:41:03.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-03-09T16:41:03.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-03-09T16:41:03.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T16:41:03.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-03-09T16:41:03.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:41:03.711 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:41:03.712 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:41:03.712 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:41:03.712 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T16:41:03.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-09T16:41:03.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: . as $in 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: | if type == "object" then 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: reduce keys[] as $key 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: else f 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr: end; 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T16:41:03.734 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T16:41:03.735 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T16:41:03.735 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T16:41:03.735 INFO:tasks.workunit.client.0.smithi146.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T16:41:03.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-09T16:41:03.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T16:41:03.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:41:03.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T16:41:03.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T16:41:03.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-03-09T16:41:03.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-09T16:41:04.137 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'csr_pool' removed 2024-03-09T16:41:04.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:41:04.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:41:04.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:41:04.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:41:04.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:41:04.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:41:04.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:41:04.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:41:04.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:41:04.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:41:04.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:41:04.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:41:04.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:41:04.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:41:04.261 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:41:04.262 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:41:04.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:41:04.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:41:04.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:41:04.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.276 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:41:04.279 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:41:04.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:41:04.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:41:04.282 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:41:04.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:41:04.282 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:41:04.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:41:04.283 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:41:04.283 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:41:04.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:41:04.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:41:04.284 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:41:04.284 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:41:04.285 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:41:04.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:41:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:41:04.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:41:04.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:41:04.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.670 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.671 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:04.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:41:04.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:41:04.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:41:04.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:41:04.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:41:04.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:41:04.863 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:41:04.864 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:41:04.944 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:04.945 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:41:04.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:41:04.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:41:05.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:41:05.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:05.830 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:05.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:05.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:41:05.831 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:41:05.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-09T16:41:05.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:41:05.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-09T16:41:05.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:41:05.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:41:05.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:41:05.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:41:05.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:41:05.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4073b10c-052c-4011-9c22-c927824d815b 2024-03-09T16:41:05.867 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 4073b10c-052c-4011-9c22-c927824d815b 2024-03-09T16:41:05.867 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4073b10c-052c-4011-9c22-c927824d815b' 2024-03-09T16:41:05.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:41:05.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhkexlFd9rNBAA+1wwnXWdBeD8LChMP4tFnw== 2024-03-09T16:41:05.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhkexlFd9rNBAA+1wwnXWdBeD8LChMP4tFnw=="}' 2024-03-09T16:41:05.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4073b10c-052c-4011-9c22-c927824d815b -i td/osd-scrub-repair/0/new.json 2024-03-09T16:41:06.161 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:41:06.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:41:06.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhkexlFd9rNBAA+1wwnXWdBeD8LChMP4tFnw== --osd-uuid 4073b10c-052c-4011-9c22-c927824d815b 2024-03-09T16:41:06.202 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.201+0000 7f7d77c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:06.212 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.210+0000 7f7d77c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:06.213 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.212+0000 7f7d77c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:06.213 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.212+0000 7f7d77c0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:41:06.213 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.212+0000 7f7d77c0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:41:06.213 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.212+0000 7f7d77c0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:41:06.303 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:06.302+0000 7f7d77c0e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:41:11.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:41:11.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:41:11.911 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:41:11.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:41:11.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:41:12.267 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:41:12.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:41:12.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:41:12.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:41:12.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:41:12.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:41:12.294 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:12.293+0000 7fc2eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:12.319 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:12.318+0000 7fc2eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:12.320 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:12.319+0000 7fc2eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:12.581 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:41:12.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:41:12.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:12.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:12.852 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:12.858 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:12.857+0000 7fc2eee0f640 -1 Falling back to public interface 2024-03-09T16:41:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:13.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:13.854 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:41:13.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:41:13.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:13.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:14.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:15.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:15.142 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:15.142 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:41:15.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:41:15.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:15.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:15.198 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:15.193+0000 7fc2eee0f640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:41:15.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:16.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:16.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:16.462 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:41:16.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:41:16.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:16.462 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:16.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:17.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:17.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:17.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:41:17.749 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:41:17.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:17.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:18.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:19.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:19.129 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:41:19.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:41:19.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:19.130 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:19.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:20.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:20.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:20.415 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:41:20.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:41:20.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:20.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:20.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:21.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:21.706 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:21.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:22.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:23.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:23.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:23.125 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:41:23.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:41:23.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:23.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:23.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:24.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:24.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:24.403 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:41:24.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:41:24.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:24.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:24.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:25.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:25.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:25.712 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:41:25.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:41:25.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:25.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:25.994 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:26.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:26.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:26.996 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:41:26.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:41:26.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:26.996 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:27.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:28.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:28.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:28.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:41:28.580 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:41:28.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:28.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:29.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:30.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:30.037 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:30.037 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:41:30.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:41:30.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:30.038 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:30.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:31.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:31.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:31.623 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:41:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:41:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:31.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:32.019 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:33.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:33.021 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:41:33.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:41:33.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:33.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:33.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:34.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:34.635 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:34.635 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:41:34.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:41:34.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:34.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:35.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:36.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:36.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:36.324 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:41:36.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:41:36.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:36.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:36.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:37.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:37.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:37.902 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:41:37.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:41:37.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:37.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:41:38.179 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3939270649,v1:127.0.0.1:6803/3939270649] [v2:127.0.0.1:6804/3939270649,v1:127.0.0.1:6805/3939270649] exists,up 4073b10c-052c-4011-9c22-c927824d815b 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:41:38.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:41:38.184 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:41:38.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:41:38.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:41:38.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a68dccb1-bad9-4f97-9f11-66bdd099c559 2024-03-09T16:41:38.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a68dccb1-bad9-4f97-9f11-66bdd099c559' 2024-03-09T16:41:38.186 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 a68dccb1-bad9-4f97-9f11-66bdd099c559 2024-03-09T16:41:38.187 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:41:38.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCkexlkH/wCxAAS3LBXkmdAeeat3uW9Vh+kw== 2024-03-09T16:41:38.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCkexlkH/wCxAAS3LBXkmdAeeat3uW9Vh+kw=="}' 2024-03-09T16:41:38.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a68dccb1-bad9-4f97-9f11-66bdd099c559 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:41:38.532 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:41:38.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:41:38.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCkexlkH/wCxAAS3LBXkmdAeeat3uW9Vh+kw== --osd-uuid a68dccb1-bad9-4f97-9f11-66bdd099c559 2024-03-09T16:41:38.572 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.570+0000 7f34a120c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:38.617 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.615+0000 7f34a120c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:38.618 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.616+0000 7f34a120c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:38.618 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.616+0000 7f34a120c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:41:38.619 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.616+0000 7f34a120c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:41:38.619 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.616+0000 7f34a120c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:41:38.712 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:38.710+0000 7f34a120c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:41:44.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:41:44.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:41:44.689 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:41:44.689 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:41:44.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:41:45.041 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:41:45.042 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:41:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:41:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:41:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:41:45.042 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:41:45.069 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:45.067+0000 7fbf9de0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:45.105 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:45.103+0000 7fbf9de0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:45.106 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:45.104+0000 7fbf9de0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:41:45.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:41:45.316 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:41:45.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:45.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:45.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:45.637 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:45.635+0000 7fbf9de0c640 -1 Falling back to public interface 2024-03-09T16:41:46.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:46.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:46.589 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:41:46.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:41:46.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:46.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:46.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:47.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:47.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:47.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:41:47.861 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:41:47.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:47.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:47.996 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:41:47.994+0000 7fbf9de0c640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:41:48.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:49.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:49.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:49.147 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:41:49.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:41:49.147 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:49.148 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:49.506 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:50.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:50.507 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:50.508 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:41:50.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:41:50.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:50.509 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:50.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:51.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:51.825 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:51.825 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:41:51.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:41:51.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:51.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:52.105 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:53.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:53.106 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:53.107 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:41:53.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:41:53.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:53.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:53.385 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:54.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:54.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:54.386 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:41:54.386 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:41:54.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:54.387 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:54.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:55.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:55.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:55.765 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:41:55.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:41:55.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:55.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:56.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:57.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:57.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:57.077 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:41:57.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:41:57.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:57.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:57.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:58.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:58.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:58.420 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:41:58.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:41:58.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:58.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:41:58.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:41:59.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:41:59.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:41:59.747 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:41:59.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:41:59.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:41:59.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:00.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:01.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:01.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:01.030 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:42:01.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:42:01.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:01.031 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:01.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:02.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:02.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:02.613 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:42:02.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:42:02.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:02.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:03.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:04.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:04.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:04.181 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:42:04.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:42:04.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:04.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:04.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:05.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:05.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:05.882 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:42:05.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:42:05.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:05.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:06.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:07.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:07.768 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:42:07.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:42:07.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:07.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:08.551 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:09.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:09.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:09.553 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:42:09.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:42:09.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:09.553 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:10.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:11.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:11.099 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:42:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:42:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:42:11.408 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/286363326,v1:127.0.0.1:6811/286363326] [v2:127.0.0.1:6812/286363326,v1:127.0.0.1:6813/286363326] exists,up a68dccb1-bad9-4f97-9f11-66bdd099c559 2024-03-09T16:42:11.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:42:11.413 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:42:11.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:42:11.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:42:11.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:42:11.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84ee6b9e-addc-43a3-bce6-188072c354e4 2024-03-09T16:42:11.418 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 84ee6b9e-addc-43a3-bce6-188072c354e4 2024-03-09T16:42:11.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 84ee6b9e-addc-43a3-bce6-188072c354e4' 2024-03-09T16:42:11.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:42:11.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjkexlQO15GRAALNi2GxyLvkoikefWUKdqYg== 2024-03-09T16:42:11.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjkexlQO15GRAALNi2GxyLvkoikefWUKdqYg=="}' 2024-03-09T16:42:11.429 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84ee6b9e-addc-43a3-bce6-188072c354e4 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:42:11.798 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:42:11.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:42:11.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjkexlQO15GRAALNi2GxyLvkoikefWUKdqYg== --osd-uuid 84ee6b9e-addc-43a3-bce6-188072c354e4 2024-03-09T16:42:11.839 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.836+0000 7fd7b6b21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:11.882 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.880+0000 7fd7b6b21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:11.882 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.881+0000 7fd7b6b21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:11.883 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.882+0000 7fd7b6b21640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:42:11.883 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.882+0000 7fd7b6b21640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:42:11.883 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.882+0000 7fd7b6b21640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:42:11.977 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:11.976+0000 7fd7b6b21640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:42:18.006 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:42:18.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:42:18.007 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:42:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:42:18.008 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:42:18.360 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:42:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:42:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:42:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:42:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:42:18.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:42:18.386 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:18.385+0000 7f6736c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:18.388 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:18.387+0000 7f6736c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:18.389 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:18.388+0000 7f6736c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:42:18.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:42:18.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:42:18.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:42:18.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:42:18.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:42:18.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:18.633 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:42:18.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:42:18.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:18.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:18.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:18.916 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:18.915+0000 7f6736c15640 -1 Falling back to public interface 2024-03-09T16:42:19.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:19.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:19.904 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:42:19.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:42:19.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:19.904 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:20.183 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:21.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:21.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:21.185 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:42:21.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:42:21.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:21.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:21.379 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:42:21.378+0000 7f6736c15640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:42:21.486 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:22.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:22.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:22.487 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:42:22.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:42:22.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:22.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:22.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:23.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:23.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:23.770 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:42:23.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:42:23.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:23.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:24.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:25.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:25.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:25.054 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:42:25.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:42:25.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:25.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:25.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:26.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:26.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:26.403 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:42:26.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:42:26.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:26.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:26.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:27.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:27.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:27.685 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:42:27.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:42:27.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:27.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:28.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:29.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:29.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:29.030 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:42:29.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:42:29.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:29.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:29.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:30.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:30.404 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:30.404 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:42:30.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:42:30.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:30.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:30.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:31.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:31.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:31.685 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:42:31.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:42:31.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:31.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:32.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:33.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:33.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:33.029 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:42:33.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:42:33.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:33.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:33.572 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:34.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:34.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:34.575 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:42:34.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:42:34.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:34.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:35.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:36.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:36.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:36.402 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:42:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:42:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:36.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:36.911 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:37.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:37.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:37.913 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:42:37.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:42:37.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:37.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:38.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:39.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:39.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:39.383 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:42:39.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:42:39.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:39.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:40.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:41.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:41.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:41.099 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:42:41.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:42:41.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:41.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:41.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:42.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:42.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:42.646 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:42:42.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:42:42.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:42.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:42.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:42:43.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:42:43.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:42:43.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:42:43.988 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:42:43.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:42:43.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:42:44.293 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3688920618,v1:127.0.0.1:6819/3688920618] [v2:127.0.0.1:6820/3688920618,v1:127.0.0.1:6821/3688920618] exists,up 84ee6b9e-addc-43a3-bce6-188072c354e4 2024-03-09T16:42:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:42:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:42:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:42:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-09T16:42:44.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:42:44.596 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:42:44.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:42:44.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:42:45.049 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:42:45.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:42:46.061 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:42:48.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-09T16:42:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:42:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:42:48.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:42:48.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:42:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:42:48.156 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:42:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:42:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:42:48.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:42:48.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:42:48.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:42:48.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:42:48.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:42:48.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:42:48.259 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:48.531 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:42:48.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T16:42:48.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T16:42:48.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T16:42:48.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:48.634 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:42:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T16:42:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T16:42:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-03-09T16:42:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:48.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:42:48.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-09T16:42:48.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-09T16:42:48.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509448' 2024-03-09T16:42:48.841 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:48.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T16:42:48.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:48.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:42:48.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T16:42:48.843 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:48.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T16:42:48.844 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836502 2024-03-09T16:42:48.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T16:42:48.845 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:42:49.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T16:42:49.118 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:42:50.119 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:42:50.120 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:42:50.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-09T16:42:50.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:50.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T16:42:50.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:50.405 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:42:50.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T16:42:50.406 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:50.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T16:42:50.407 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672975 2024-03-09T16:42:50.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T16:42:50.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:42:50.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T16:42:50.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:50.681 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-09T16:42:50.682 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:50.682 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:42:50.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-09T16:42:50.683 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:50.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-09T16:42:50.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-09T16:42:50.685 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509448 2024-03-09T16:42:50.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:42:50.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-03-09T16:42:51.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:42:51.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:42:51.006 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:42:51.438 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:42:51.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:42:51.439 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:42:51.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:42:51.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:42:51.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:42:51.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:42:51.440 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:42:51.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:42:51.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:42:51.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:42:51.719 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:42:52.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:42:52.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T16:42:52.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:42:52.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:42:53.114 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:42:53.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:42:54.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T16:42:54.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:42:54.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:42:54.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:42:54.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:42:54.128 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:42:54.230 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:54.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:42:54.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-09T16:42:54.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-09T16:42:54.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-09T16:42:54.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:54.606 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:42:54.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T16:42:54.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T16:42:54.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-03-09T16:42:54.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:42:54.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509450' 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-09T16:42:54.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:54.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:42:54.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-09T16:42:54.816 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:54.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-09T16:42:54.817 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836504 2024-03-09T16:42:54.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-09T16:42:54.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:42:55.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-09T16:42:55.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:42:56.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:42:56.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:42:56.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-09T16:42:56.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:56.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T16:42:56.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:56.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:42:56.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T16:42:56.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:56.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T16:42:56.381 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672977 2024-03-09T16:42:56.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T16:42:56.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:42:56.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T16:42:56.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:42:56.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T16:42:56.655 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:42:56.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:42:56.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T16:42:56.656 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:42:56.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T16:42:56.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T16:42:56.658 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509450 2024-03-09T16:42:56.658 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:42:56.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509450 2024-03-09T16:42:56.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:42:56.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:42:56.928 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:42:57.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:42:57.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:42:57.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:42:57.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:42:57.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:42:57.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:42:57.560 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:42:57.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:42:57.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:42:57.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:42:57.899 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:42:58.327 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:42:58.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:42:58.804 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:42:58.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:42:58.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:42:58.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:42:59.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-09T16:42:59.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:42:59.076 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-09T16:42:59.076 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-09T16:42:59.076 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:42:59.357 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T16:42:59.357 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:42:59.357 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:42:59.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:42:59.749 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:42:59.760 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:43:00.219 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:43:00.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:43:00.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:43:00.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:43:00.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-09T16:43:00.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:43:00.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-09T16:43:00.506 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-09T16:43:00.506 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:43:00.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T16:43:00.784 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:43:00.784 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:43:00.784 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T16:43:00.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-09T16:43:00.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-09T16:43:00.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:43:00.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:43:01.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:43:01.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:01.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:43:01.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:43:01.094 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:43:01.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:43:01.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:43:01.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:43:02.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:43:02.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:02.402 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:43:02.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:43:02.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:43:02.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:43:02.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:43:02.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:43:02.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:43:02.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-09T16:43:02.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-09T16:43:03.811 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-09T16:43:04.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-09T16:43:04.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-09T16:43:05.482 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-09T16:43:05.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-09T16:43:05.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-09T16:43:06.776 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-09T16:43:07.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-09T16:43:07.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-09T16:43:08.415 INFO:tasks.workunit.client.0.smithi146.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-09T16:43:08.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:43:08.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:08.934 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:43:08.935 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:43:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:43:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:43:08.936 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:43:08.937 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:43:08.963 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:08.962+0000 7fc406cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:08.965 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:08.964+0000 7fc406cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:08.966 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:08.965+0000 7fc406cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:09.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:43:09.218 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:43:09.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:09.218 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:09.483 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:09.482+0000 7fc406cf5640 -1 Falling back to public interface 2024-03-09T16:43:09.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:10.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:10.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:10.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:43:10.490 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:43:10.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:10.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:10.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:11.768 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:11.877 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:11.876+0000 7fc406cf5640 -1 osd.0 34 log_to_monitors true 2024-03-09T16:43:12.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:43:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:43:13.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:13.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:13.336 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/2635843649,v1:127.0.0.1:6803/2635843649] [v2:127.0.0.1:6804/2635843649,v1:127.0.0.1:6805/2635843649] exists,up 4073b10c-052c-4011-9c22-c927824d815b 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:43:13.337 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:13.338 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:43:13.339 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:43:13.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:43:13.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:13.360+0000 7fae9041e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:13.363 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:13.362+0000 7fae9041e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:13.364 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:13.363+0000 7fae9041e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:13.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:43:13.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:43:13.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:13.611 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:43:13.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:13.885 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:13.884+0000 7fae9041e640 -1 Falling back to public interface 2024-03-09T16:43:14.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:14.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:14.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:43:14.879 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:43:14.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:14.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:43:15.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:16.188 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:16.186+0000 7fae9041e640 -1 osd.1 36 log_to_monitors true 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:16.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:43:16.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:17.469 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:17.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:17.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:43:17.470 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:43:17.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:17.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:43:17.753 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/2575184597,v1:127.0.0.1:6811/2575184597] [v2:127.0.0.1:6812/2575184597,v1:127.0.0.1:6813/2575184597] exists,up a68dccb1-bad9-4f97-9f11-66bdd099c559 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:43:17.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:17.755 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:43:17.756 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:43:17.757 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:43:17.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:43:17.757 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:43:17.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:43:17.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:43:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:43:17.758 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:43:17.778 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:17.777+0000 7fe545601640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:17.780 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:17.779+0000 7fe545601640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:17.781 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:17.780+0000 7fe545601640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:18.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:43:18.300 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:18.299+0000 7fe545601640 -1 Falling back to public interface 2024-03-09T16:43:18.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:19.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:43:19.586 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:20.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:20.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:20.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:43:20.588 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:43:20.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:20.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:43:20.656 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:20.655+0000 7fe545601640 -1 osd.2 38 log_to_monitors true 2024-03-09T16:43:20.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:21.866 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:43:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/241154723,v1:127.0.0.1:6819/241154723] [v2:127.0.0.1:6820/241154723,v1:127.0.0.1:6821/241154723] exists,up 84ee6b9e-addc-43a3-bce6-188072c354e4 2024-03-09T16:43:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:43:22.216 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:43:22.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:43:22.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-09T16:43:22.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:43:22.580 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' removed 2024-03-09T16:43:22.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:43:22.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:43:23.016 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:43:23.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:43:24.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:43:25.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-09T16:43:25.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:43:25.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:43:25.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:43:25.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:43:25.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:43:25.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:43:25.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:43:25.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:43:25.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:43:25.752 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:43:26.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:43:26.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-03-09T16:43:26.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-03-09T16:43:26.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-03-09T16:43:26.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:43:26.140 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:43:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-03-09T16:43:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-03-09T16:43:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-03-09T16:43:26.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:43:26.244 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:43:26.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-09T16:43:26.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-09T16:43:26.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-201863462915' 2024-03-09T16:43:26.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:43:26.346 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-03-09T16:43:26.346 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:43:26.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:43:26.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-03-09T16:43:26.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:43:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-03-09T16:43:26.349 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 176093659140 2024-03-09T16:43:26.349 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-03-09T16:43:26.350 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:43:26.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-03-09T16:43:26.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:43:27.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:43:27.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:43:27.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-03-09T16:43:27.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:43:28.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:43:28.906 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:43:29.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659140 2024-03-09T16:43:29.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:43:29.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-03-09T16:43:29.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:43:29.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:43:29.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-03-09T16:43:29.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:43:29.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-03-09T16:43:29.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-03-09T16:43:29.224 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 188978561027 2024-03-09T16:43:29.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:43:29.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561027 2024-03-09T16:43:29.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:43:29.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-09T16:43:29.496 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:43:29.497 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:43:29.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-09T16:43:29.497 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:43:29.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-09T16:43:29.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-09T16:43:29.499 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 201863462915 2024-03-09T16:43:29.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:43:29.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-09T16:43:29.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:43:29.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:43:29.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:43:30.113 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:43:30.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:43:30.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:43:30.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:43:30.114 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:43:30.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:43:30.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:43:30.115 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:43:30.399 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:43:30.399 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:43:30.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:43:30.400 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:43:30.737 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-09T16:43:30.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-09T16:43:30.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:43:31.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-09T16:43:31.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-09T16:43:31.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:43:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:43:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:31.064 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:31.331 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:43:31.587 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:43:31.599 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:31.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:31.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:31.600 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:31.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:42:52.954098+0000 '>' 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:31.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:43:32.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:43:32.865 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:43:32.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:43:32.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:32.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:32.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:32.866 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:33.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:42:52.954098+0000 '>' 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:33.149 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:43:34.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:43:34.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:43:34.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:43:34.154 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:34.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:34.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:34.155 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:34.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:42:52.954098+0000 '>' 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:34.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:43:35.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:43:35.427 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:43:35.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:43:35.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:35.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:35.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:35.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:35.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:42:52.954098+0000 '>' 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:35.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:43:36.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:43:36.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:43:36.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:43:36.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:43:36.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:43:36.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:43:36.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:43:36.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:43:31.924501+0000 '>' 2024-03-09T16:42:52.954098+0000 2024-03-09T16:43:36.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:43:36.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-09T16:43:36.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-09T16:43:36.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-09T16:43:36.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-09T16:43:36.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-09T16:43:36.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-09T16:43:37.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-09T16:43:37.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-09T16:43:37.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-09T16:43:37.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:43:37.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:43:37.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:43:37.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:43:37.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:43:37.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:43:37.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:43:37.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:43:37.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:43:37.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:43:37.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:43:37.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:43:37.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:43:37.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:43:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:43:37.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:43:37.224 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:43:37.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.225 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:43:37.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:43:37.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:43:37.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:43:37.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:43:37.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:43:37.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:43:37.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:43:37.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:43:37.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:43:37.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:43:37.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:43:37.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:43:37.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:43:37.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:43:37.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:43:37.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:43:37.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:43:37.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:43:37.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:43:37.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.251 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-03-09T16:43:37.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-09T16:43:37.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T16:43:37.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:43:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:43:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:43:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:43:37.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:43:37.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:43:37.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:37.608 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:37.609 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:37.609 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.609 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.609 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:37.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:43:37.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:43:37.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:43:37.794 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:43:37.795 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:43:37.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:43:37.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:43:37.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:37.878 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:37.879 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:43:37.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:43:37.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:43:37.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T16:43:37.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:43:37.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:43:37.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:43:37.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:43:37.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:43:37.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:38.770 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:38.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:38.771 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:43:38.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:43:38.797 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:43:38.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:43:38.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:43:38.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:43:38.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:43:38.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:43:38.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd0923ed-b756-470c-b76d-5d899d057b40 2024-03-09T16:43:38.809 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 bd0923ed-b756-470c-b76d-5d899d057b40 2024-03-09T16:43:38.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bd0923ed-b756-470c-b76d-5d899d057b40' 2024-03-09T16:43:38.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:43:38.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6kexlaxTNMBAApeTJX2uf6JTtUgrqp4xXKA== 2024-03-09T16:43:38.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6kexlaxTNMBAApeTJX2uf6JTtUgrqp4xXKA=="}' 2024-03-09T16:43:38.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd0923ed-b756-470c-b76d-5d899d057b40 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:43:39.049 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:43:39.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:43:39.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6kexlaxTNMBAApeTJX2uf6JTtUgrqp4xXKA== --osd-uuid bd0923ed-b756-470c-b76d-5d899d057b40 2024-03-09T16:43:39.089 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.088+0000 7faeaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:39.100 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.099+0000 7faeaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:39.101 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.100+0000 7faeaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:39.102 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.101+0000 7faeaa8d9640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:43:39.102 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.101+0000 7faeaa8d9640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:43:39.102 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.101+0000 7faeaa8d9640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:43:39.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:39.191+0000 7faeaa8d9640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:43:44.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:43:44.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:43:44.893 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:43:44.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:43:44.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:43:45.243 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:43:45.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:43:45.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:43:45.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:43:45.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:43:45.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:43:45.269 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:45.268+0000 7fb7eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:45.271 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:45.270+0000 7fb7eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:45.272 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:45.271+0000 7fb7eee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:43:45.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:43:45.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:43:45.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:43:45.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:45.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:45.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:45.799 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:45.798+0000 7fb7eee0f640 -1 Falling back to public interface 2024-03-09T16:43:46.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:46.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:46.789 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:43:46.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:43:46.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:46.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:47.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:48.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:48.062 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:48.063 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:43:48.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:43:48.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:48.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:48.148 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:43:48.146+0000 7fb7eee0f640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:43:48.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:49.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:49.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:49.351 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:43:49.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:43:49.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:49.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:49.634 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:50.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:50.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:50.636 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:43:50.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:43:50.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:50.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:50.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:51.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:51.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:51.921 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:43:51.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:43:51.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:51.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:52.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:53.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:53.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:53.206 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:43:53.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:43:53.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:53.207 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:53.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:54.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:54.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:54.489 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:43:54.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:43:54.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:54.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:54.770 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:55.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:55.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:55.771 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:43:55.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:43:55.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:55.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:56.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:57.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:57.084 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:43:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:43:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:57.085 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:57.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:58.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:58.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:58.367 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:43:58.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:43:58.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:58.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:43:58.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:43:59.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:43:59.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:43:59.695 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:43:59.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:43:59.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:43:59.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:00.050 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:01.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:01.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:01.052 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:44:01.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:44:01.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:01.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:01.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:02.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:02.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:02.523 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:44:02.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:44:02.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:02.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:03.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:04.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:04.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:04.103 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:44:04.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:44:04.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:04.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:04.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:05.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:05.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:05.384 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:44:05.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:44:05.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:05.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:06.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:07.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:07.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:07.145 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:44:07.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:44:07.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:07.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:07.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:08.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:08.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:08.915 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:44:08.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:44:08.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:08.916 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:09.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:10.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:10.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:10.675 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:44:10.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:44:10.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:10.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:44:10.950 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/615994433,v1:127.0.0.1:6803/615994433] [v2:127.0.0.1:6804/615994433,v1:127.0.0.1:6805/615994433] exists,up bd0923ed-b756-470c-b76d-5d899d057b40 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:44:10.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:44:10.956 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b721c2cb-33b6-418c-bda5-f007ad423cea 2024-03-09T16:44:10.957 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 b721c2cb-33b6-418c-bda5-f007ad423cea 2024-03-09T16:44:10.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b721c2cb-33b6-418c-bda5-f007ad423cea' 2024-03-09T16:44:10.958 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:44:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDakexlsz7kORAAxsmHSEa7QgoXLXMEAat8cg== 2024-03-09T16:44:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDakexlsz7kORAAxsmHSEa7QgoXLXMEAat8cg=="}' 2024-03-09T16:44:10.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b721c2cb-33b6-418c-bda5-f007ad423cea -i td/osd-scrub-repair/1/new.json 2024-03-09T16:44:11.292 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:44:11.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:44:11.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDakexlsz7kORAAxsmHSEa7QgoXLXMEAat8cg== --osd-uuid b721c2cb-33b6-418c-bda5-f007ad423cea 2024-03-09T16:44:11.332 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.331+0000 7fe1f5e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:11.376 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.375+0000 7fe1f5e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:11.377 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.376+0000 7fe1f5e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:11.377 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.377+0000 7fe1f5e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:44:11.377 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.377+0000 7fe1f5e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:44:11.378 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.377+0000 7fe1f5e11640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:44:11.451 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:11.450+0000 7fe1f5e11640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:44:17.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:44:17.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:44:17.448 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:44:17.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:44:17.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:44:17.799 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:44:17.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:44:17.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:44:17.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:44:17.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:44:17.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:44:17.825 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:17.824+0000 7fe437e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:17.827 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:17.826+0000 7fe437e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:17.828 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:17.827+0000 7fe437e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:44:18.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:18.074 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:44:18.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:44:18.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:18.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:18.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:18.349 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:18.349+0000 7fe437e0c640 -1 Falling back to public interface 2024-03-09T16:44:19.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:19.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:19.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:44:19.342 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:44:19.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:19.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:19.615 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:20.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:20.617 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:20.617 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:44:20.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:44:20.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:20.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:20.672 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:20.671+0000 7fe437e0c640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:44:20.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:21.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:21.892 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:21.892 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:44:21.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:44:21.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:21.893 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:22.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:23.173 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:23.174 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:23.174 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:44:23.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:44:23.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:23.175 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:23.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:24.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:24.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:24.504 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:44:24.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:44:24.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:24.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:24.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:25.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:25.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:25.786 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:44:25.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:44:25.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:25.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:26.161 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:27.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:27.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:27.163 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:44:27.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:44:27.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:27.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:27.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:28.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:28.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:28.534 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:44:28.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:44:28.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:28.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:28.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:29.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:29.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:29.819 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:44:29.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:44:29.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:29.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:30.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:31.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:31.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:31.128 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:44:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:44:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:31.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:31.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:32.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:32.483 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:32.483 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:44:32.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:44:32.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:32.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:32.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:33.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:33.766 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:33.766 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:44:33.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:44:33.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:33.767 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:34.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:35.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:35.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:35.088 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:44:35.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:44:35.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:35.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:35.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:36.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:36.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:36.594 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:44:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:44:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:36.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:37.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:38.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:38.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:38.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:44:38.069 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:44:38.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:38.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:38.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:39.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:39.721 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:39.722 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:44:39.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:44:39.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:39.722 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:40.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:41.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:43.138 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:43.138 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:43.139 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:43.140 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:44:43.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:44:43.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:43.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:43.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:44.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:44.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:44.292 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:44:44.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:44:44.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:44.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:44:44.571 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3835551315,v1:127.0.0.1:6811/3835551315] [v2:127.0.0.1:6812/3835551315,v1:127.0.0.1:6813/3835551315] exists,up b721c2cb-33b6-418c-bda5-f007ad423cea 2024-03-09T16:44:44.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:44:44.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:44:44.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:44:44.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:44:44.581 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:44:44.582 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:44:44.583 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57ce5ac9-6363-411e-955e-d1335e3482c0 2024-03-09T16:44:44.584 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 57ce5ac9-6363-411e-955e-d1335e3482c0 2024-03-09T16:44:44.584 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 57ce5ac9-6363-411e-955e-d1335e3482c0' 2024-03-09T16:44:44.584 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:44:44.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8kexleYxJIxAA/Du1Edh2UqC8C5RYuWRQJA== 2024-03-09T16:44:44.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8kexleYxJIxAA/Du1Edh2UqC8C5RYuWRQJA=="}' 2024-03-09T16:44:44.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57ce5ac9-6363-411e-955e-d1335e3482c0 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:44:44.959 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:44:44.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:44:44.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8kexleYxJIxAA/Du1Edh2UqC8C5RYuWRQJA== --osd-uuid 57ce5ac9-6363-411e-955e-d1335e3482c0 2024-03-09T16:44:44.999 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:44.998+0000 7f762e10d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:45.043 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.043+0000 7f762e10d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:45.044 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.044+0000 7f762e10d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:45.045 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.044+0000 7f762e10d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:44:45.045 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.044+0000 7f762e10d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:44:45.045 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.044+0000 7f762e10d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:44:45.127 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:45.126+0000 7f762e10d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:44:51.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:44:51.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:44:51.169 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:44:51.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:44:51.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:44:51.520 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:44:51.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:44:51.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:44:51.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:44:51.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:44:51.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:44:51.546 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:51.546+0000 7fa089c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:51.548 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:51.547+0000 7fa089c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:51.549 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:51.548+0000 7fa089c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:44:51.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:44:51.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:44:51.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:44:51.793 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:44:51.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:44:51.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:51.794 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:44:51.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:44:51.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:51.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:52.064 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:52.080 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:52.080+0000 7fa089c0b640 -1 Falling back to public interface 2024-03-09T16:44:53.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:53.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:53.066 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:44:53.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:44:53.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:53.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:53.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:54.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:54.346 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:54.347 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:44:54.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:44:54.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:54.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:54.418 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:44:54.415+0000 7fa089c0b640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:44:54.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:55.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:55.642 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:44:55.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:55.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:44:55.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:55.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:55.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:56.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:56.947 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:56.947 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:44:56.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:44:56.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:56.948 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:57.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:58.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:58.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:58.280 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:44:58.280 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:44:58.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:58.281 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:58.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:44:59.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:44:59.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:44:59.661 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:44:59.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:44:59.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:44:59.662 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:44:59.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:00.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:01.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:01.097 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:45:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:45:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:01.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:01.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:02.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:02.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:02.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:45:02.237 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:45:02.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:02.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:02.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:03.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:03.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:03.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:45:03.519 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:45:03.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:03.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:03.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:04.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:04.802 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:45:04.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:04.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:45:04.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:04.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:05.112 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:06.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:06.114 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:06.114 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:45:06.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:45:06.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:06.115 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:06.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:07.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:07.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:07.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:45:07.641 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:45:07.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:07.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:08.055 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:09.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:09.056 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:45:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:45:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:09.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:09.882 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:10.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:10.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:10.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:45:10.884 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:45:10.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:10.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:11.392 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:12.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:12.394 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:12.394 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:45:12.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:45:12.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:12.395 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:12.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:13.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:13.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:13.978 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:45:13.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:45:13.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:13.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:14.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:15.533 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:15.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:15.534 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:45:15.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:45:15.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:15.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:15.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:16.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:45:16.942 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:45:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:16.943 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:17.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:18.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:18.258 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:45:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:45:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:18.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:18.537 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/655908673,v1:127.0.0.1:6819/655908673] [v2:127.0.0.1:6820/655908673,v1:127.0.0.1:6821/655908673] exists,up 57ce5ac9-6363-411e-955e-d1335e3482c0 2024-03-09T16:45:18.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:45:18.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:45:18.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:45:18.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-09T16:45:18.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:45:18.838 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:45:18.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:45:18.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:45:19.265 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:45:19.277 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:45:20.278 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:45:22.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-09T16:45:22.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:45:22.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:45:22.403 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:45:22.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:45:22.404 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:45:22.404 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:45:22.404 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:45:22.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:45:22.405 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:45:22.505 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:22.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:45:22.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T16:45:22.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T16:45:22.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T16:45:22.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:22.881 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:45:22.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705677 2024-03-09T16:45:22.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705677 2024-03-09T16:45:22.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-38654705677' 2024-03-09T16:45:22.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:22.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:45:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542153 2024-03-09T16:45:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542153 2024-03-09T16:45:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-38654705677 2-60129542153' 2024-03-09T16:45:23.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:23.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T16:45:23.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:23.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:45:23.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T16:45:23.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:23.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T16:45:23.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T16:45:23.101 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T16:45:23.101 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:45:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T16:45:23.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:45:24.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:45:24.373 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:45:24.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T16:45:24.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:45:25.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T16:45:25.649 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:45:25.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T16:45:25.958 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:25.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705677 2024-03-09T16:45:25.959 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:25.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:45:25.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705677 2024-03-09T16:45:25.962 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:25.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705677 2024-03-09T16:45:25.962 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 38654705677 2024-03-09T16:45:25.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2024-03-09T16:45:25.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:45:26.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705678 -lt 38654705677 2024-03-09T16:45:26.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:26.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542153 2024-03-09T16:45:26.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:26.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:45:26.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542153 2024-03-09T16:45:26.236 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:26.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542153 2024-03-09T16:45:26.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2024-03-09T16:45:26.237 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542153 2024-03-09T16:45:26.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:45:26.508 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542153 -lt 60129542153 2024-03-09T16:45:26.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:45:26.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:45:26.509 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:45:26.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:45:26.848 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:45:26.849 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:45:27.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:45:27.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:45:27.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:45:27.138 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:45:27.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T16:45:27.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:45:27.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T16:45:27.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T16:45:27.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T16:45:28.652 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'ecpool' created 2024-03-09T16:45:28.663 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:45:29.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T16:45:29.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T16:45:30.176 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T16:45:30.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:45:30.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:45:30.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:45:30.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:45:30.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:45:30.282 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:30.563 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:45:30.671 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T16:45:30.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T16:45:30.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T16:45:30.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:30.672 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:45:30.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705680 2024-03-09T16:45:30.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705680 2024-03-09T16:45:30.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-38654705680' 2024-03-09T16:45:30.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:45:30.789 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:45:30.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542155 2024-03-09T16:45:30.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542155 2024-03-09T16:45:30.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-38654705680 2-60129542155' 2024-03-09T16:45:30.920 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:30.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T16:45:30.920 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:30.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:45:30.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T16:45:30.922 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:30.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T16:45:30.923 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836503 2024-03-09T16:45:30.924 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T16:45:30.924 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:45:31.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T16:45:31.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:31.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705680 2024-03-09T16:45:31.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:31.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:45:31.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705680 2024-03-09T16:45:31.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:31.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705680 2024-03-09T16:45:31.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2024-03-09T16:45:31.198 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 38654705680 2024-03-09T16:45:31.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:45:31.471 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705680 -lt 38654705680 2024-03-09T16:45:31.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:45:31.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542155 2024-03-09T16:45:31.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:45:31.473 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:45:31.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542155 2024-03-09T16:45:31.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:45:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542155 2024-03-09T16:45:31.475 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2024-03-09T16:45:31.475 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 60129542155 2024-03-09T16:45:31.476 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:45:31.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542154 -lt 60129542155 2024-03-09T16:45:31.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:45:32.746 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:45:32.746 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:45:33.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542155 -lt 60129542155 2024-03-09T16:45:33.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:45:33.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:45:33.027 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:45:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:45:33.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:45:33.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:45:33.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:45:33.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:45:33.650 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:45:34.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:45:34.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-09T16:45:34.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:45:34.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:45:34.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:45:34.543 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:45:34.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:45:35.054 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:45:35.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:45:35.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:45:35.065 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:45:35.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-09T16:45:35.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:45:35.365 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-09T16:45:35.365 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-09T16:45:35.365 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:45:35.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T16:45:35.696 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:35.696 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:45:35.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T16:45:35.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T16:45:35.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T16:45:35.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T16:45:36.176 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is set 2024-03-09T16:45:36.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T16:45:36.568 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:45:36.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:45:36.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:45:36.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:45:36.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-09T16:45:36.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T16:45:36.848 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-09T16:45:36.848 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-09T16:45:36.849 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr:0' 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-09T16:45:37.125 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-09T16:45:37.126 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-09T16:45:37.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:37.126 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:45:37.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:45:37.127 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:45:37.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:45:37.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:45:37.127 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:45:37.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:45:38.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:45:38.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:38.744 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T16:45:38.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:45:38.745 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:45:38.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:45:38.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:45:38.745 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:45:39.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:45:39.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-09T16:45:39.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-09T16:45:40.218 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-09T16:45:40.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-09T16:45:40.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-09T16:45:41.797 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-09T16:45:42.311 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-09T16:45:42.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-09T16:45:43.133 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-09T16:45:43.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-09T16:45:43.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-09T16:45:44.821 INFO:tasks.workunit.client.0.smithi146.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-09T16:45:45.335 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:45:45.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:45:45.340 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:45:45.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:45:45.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:45:45.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:45:45.369 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:45.368+0000 7f351e31d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:45.371 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:45.371+0000 7f351e31d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:45.373 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:45.372+0000 7f351e31d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:45.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:45:45.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:45.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:45:45.896 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:45.895+0000 7f351e31d640 -1 Falling back to public interface 2024-03-09T16:45:45.925 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:46.927 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:45:47.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:45:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:45:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:48.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:45:48.318 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:48.317+0000 7f351e31d640 -1 osd.0 34 log_to_monitors true 2024-03-09T16:45:48.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:49.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:45:49.817 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/3230264957,v1:127.0.0.1:6803/3230264957] [v2:127.0.0.1:6804/3230264957,v1:127.0.0.1:6805/3230264957] exists,up bd0923ed-b756-470c-b76d-5d899d057b40 2024-03-09T16:45:49.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:45:49.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:45:49.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.1 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:45:49.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:45:49.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:45:49.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:45:49.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:45:49.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:45:49.843 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:49.842+0000 7f1a16411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:49.845 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:49.844+0000 7f1a16411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:49.846 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:49.845+0000 7f1a16411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:50.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:45:50.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:45:50.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:45:50.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:50.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:45:50.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:50.360+0000 7f1a16411640 -1 Falling back to public interface 2024-03-09T16:45:50.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:51.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:51.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:51.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:45:51.574 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:51.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:51.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:45:51.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:52.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:45:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:45:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:52.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:45:52.677 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:52.676+0000 7f1a16411640 -1 osd.1 35 log_to_monitors true 2024-03-09T16:45:52.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:53.989 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:45:54.299 INFO:tasks.workunit.client.0.smithi146.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [9,35) [v2:127.0.0.1:6810/1740072175,v1:127.0.0.1:6811/1740072175] [v2:127.0.0.1:6812/1740072175,v1:127.0.0.1:6813/1740072175] exists,up b721c2cb-33b6-418c-bda5-f007ad423cea 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:45:54.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:45:54.301 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T16:45:54.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.2 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T16:45:54.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:45:54.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:45:54.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:45:54.324 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:54.324+0000 7fa2fb0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:54.326 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:54.326+0000 7fa2fb0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:54.327 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:54.327+0000 7fa2fb0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:45:54.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T16:45:54.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:45:54.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:45:54.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:54.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:54.842 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:54.842+0000 7fa2fb0a4640 -1 Falling back to public interface 2024-03-09T16:45:54.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:55.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:56.153 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:57.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:57.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:57.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:45:57.156 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:45:57.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:57.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:57.282 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:45:57.281+0000 7fa2fb0a4640 -1 osd.2 37 log_to_monitors true 2024-03-09T16:45:57.468 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:45:58.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [14,37) [v2:127.0.0.1:6818/3047998465,v1:127.0.0.1:6819/3047998465] [v2:127.0.0.1:6820/3047998465,v1:127.0.0.1:6821/3047998465] exists,up 57ce5ac9-6363-411e-955e-d1335e3482c0 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-09T16:45:58.755 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:45:59.096 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' removed 2024-03-09T16:45:59.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:45:59.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:45:59.524 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:45:59.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:46:00.537 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:46:01.889 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-09T16:46:01.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:46:01.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:46:01.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:46:01.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:46:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:46:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:46:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:46:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:46:01.992 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:46:01.993 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:46:02.264 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:46:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-03-09T16:46:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-03-09T16:46:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-03-09T16:46:02.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:46:02.366 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:46:02.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-03-09T16:46:02.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-03-09T16:46:02.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-03-09T16:46:02.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:46:02.470 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-03-09T16:46:02.573 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:46:02.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:46:02.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-03-09T16:46:02.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:46:02.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-03-09T16:46:02.577 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 171798691844 2024-03-09T16:46:02.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-03-09T16:46:02.577 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:46:02.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-03-09T16:46:02.844 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:46:03.845 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:46:03.846 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:46:04.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691844 2024-03-09T16:46:04.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:46:04.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-03-09T16:46:04.163 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:46:04.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:46:04.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-03-09T16:46:04.165 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:46:04.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-03-09T16:46:04.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-03-09T16:46:04.167 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 184683593731 2024-03-09T16:46:04.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:46:04.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593731 -lt 184683593731 2024-03-09T16:46:04.445 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:46:04.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-09T16:46:04.446 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:46:04.447 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:46:04.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-09T16:46:04.448 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:46:04.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-09T16:46:04.449 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-09T16:46:04.449 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 201863462915 2024-03-09T16:46:04.450 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:46:04.720 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-09T16:46:04.720 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:46:04.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:46:04.721 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:46:05.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:46:05.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:46:05.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:46:05.347 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:46:05.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:46:05.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:46:05.348 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:46:05.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:46:05.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:46:05.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:46:05.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-09T16:46:05.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T16:46:05.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-09T16:46:05.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-09T16:46:05.686 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:46:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-09T16:46:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-09T16:46:05.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T16:46:05.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T16:46:05.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:46:05.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:46:05.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:46:05.987 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:46:06.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:06.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T16:46:06.506 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:46:06.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:46:06.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:46:06.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:46:06.523 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:46:06.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:46:06.524 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:46:06.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:45:28.525950+0000 '>' 2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:06.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:46:07.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:46:07.790 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:46:07.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:46:07.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:46:07.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:46:07.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:46:07.791 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:46:08.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:45:28.525950+0000 '>' 2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:08.094 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:46:09.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:46:09.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:46:09.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:46:09.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:46:09.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:46:09.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:46:09.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:46:09.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:46:07.319289+0000 '>' 2024-03-09T16:45:28.525950+0000 2024-03-09T16:46:09.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:46:09.372 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-09T16:46:09.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-09T16:46:09.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-09T16:46:09.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-09T16:46:09.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-09T16:46:09.374 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:46:09.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:46:09.480 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:46:09.480 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:46:09.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:46:09.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:46:09.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:46:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:46:09.589 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:46:09.589 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:46:09.590 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:46:09.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:46:09.591 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:46:09.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:46:09.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:46:09.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:46:09.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:46:09.593 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:46:09.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:46:09.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:46:09.595 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:46:09.611 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:46:09.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:09.612 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:09.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:46:09.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:46:09.614 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:46:09.620 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:46:09.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:46:09.621 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:46:09.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:46:09.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:46:09.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:46:09.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:46:09.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:09.624 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:09.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:46:09.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:46:09.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:46:09.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:46:09.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:46:09.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:09.626 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:09.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:46:09.631 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:46:09.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:46:09.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:46:09.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:46:09.631 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:46:09.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:46:09.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:46:09.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:46:09.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:46:09.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T16:46:10.010 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:46:10.010 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:10.011 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T16:46:10.194 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:46:10.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:46:10.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:46:10.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:46:10.195 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:46:10.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:46:10.197 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:46:10.277 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:10.278 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:46:10.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:46:10.279 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:46:10.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:46:10.358 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:46:11.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:46:11.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:46:11.171 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:46:11.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:46:11.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-03-09T16:46:11.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-03-09T16:46:11.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:46:11.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:46:11.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:46:11.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 2024-03-09T16:46:11.204 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28' 2024-03-09T16:46:11.205 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 2024-03-09T16:46:11.205 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:46:11.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTkuxl+jkWDRAAw4SwC9NN0Xbey9wkbRm7pw== 2024-03-09T16:46:11.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTkuxl+jkWDRAAw4SwC9NN0Xbey9wkbRm7pw=="}' 2024-03-09T16:46:11.221 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:46:11.426 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:46:11.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:46:11.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBTkuxl+jkWDRAAw4SwC9NN0Xbey9wkbRm7pw== --osd-uuid 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 2024-03-09T16:46:11.467 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.466+0000 7fa8ad11f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:11.477 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.476+0000 7fa8ad11f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:11.478 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.477+0000 7fa8ad11f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:11.478 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.478+0000 7fa8ad11f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:46:11.478 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.478+0000 7fa8ad11f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:46:11.478 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.478+0000 7fa8ad11f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:46:11.569 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:11.568+0000 7fa8ad11f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:46:17.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:46:17.134 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:46:17.135 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:46:17.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:46:17.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:46:17.487 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:46:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:46:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:46:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:46:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:46:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:46:17.515 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:17.514+0000 7f5aeff49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:17.524 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:17.523+0000 7f5aeff49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:17.525 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:17.525+0000 7f5aeff49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:46:17.789 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:46:17.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:17.789 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:18.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:18.059 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:18.058+0000 7f5aeff49640 -1 Falling back to public interface 2024-03-09T16:46:19.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:19.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:19.060 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:46:19.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:46:19.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:19.060 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:19.333 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:20.335 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:20.390 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:20.389+0000 7f5aeff49640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:46:20.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:21.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:21.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:21.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:46:21.668 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:46:21.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:21.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:21.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:22.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:22.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:46:22.952 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:46:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:22.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:23.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:24.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:24.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:24.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:46:24.328 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:46:24.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:24.329 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:24.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:25.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:25.609 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:46:25.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:25.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:46:25.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:25.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:25.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:26.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:26.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:26.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:46:26.969 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:46:26.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:26.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:27.323 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:28.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:28.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:28.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:46:28.325 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:46:28.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:28.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:28.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:29.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:29.608 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:46:29.609 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:46:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:29.960 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:30.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:30.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:30.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:46:30.962 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:46:30.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:30.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:31.319 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:32.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:32.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:32.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:46:32.321 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:46:32.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:32.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:32.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:33.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:33.656 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:46:33.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:46:33.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:33.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:34.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:35.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:35.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:35.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:46:35.270 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:46:35.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:35.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:36.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:37.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:37.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:37.072 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:46:37.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:46:37.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:37.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:37.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:38.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:38.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:38.518 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:46:38.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:46:38.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:38.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:39.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:40.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:40.143 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:46:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:46:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:40.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:40.850 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:41.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:41.853 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:41.853 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:46:41.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:46:41.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:41.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:42.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:43.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:46:43.540 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:46:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:43.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:46:43.853 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4147558233,v1:127.0.0.1:6803/4147558233] [v2:127.0.0.1:6804/4147558233,v1:127.0.0.1:6805/4147558233] exists,up 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 2024-03-09T16:46:43.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:46:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:46:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:46:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:46:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:46:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:46:43.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:46:43.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:46:43.859 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:46:43.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T16:46:43.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:46:43.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:46:43.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05e24bfa-8d87-49eb-b984-98d3ebdf35c2 2024-03-09T16:46:43.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 05e24bfa-8d87-49eb-b984-98d3ebdf35c2' 2024-03-09T16:46:43.861 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 05e24bfa-8d87-49eb-b984-98d3ebdf35c2 2024-03-09T16:46:43.862 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:46:43.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzkuxlqgIdNBAAb3c6AFFw8mQmnMMSLm9+Iw== 2024-03-09T16:46:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzkuxlqgIdNBAAb3c6AFFw8mQmnMMSLm9+Iw=="}' 2024-03-09T16:46:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05e24bfa-8d87-49eb-b984-98d3ebdf35c2 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:46:44.248 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:46:44.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:46:44.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBzkuxlqgIdNBAAb3c6AFFw8mQmnMMSLm9+Iw== --osd-uuid 05e24bfa-8d87-49eb-b984-98d3ebdf35c2 2024-03-09T16:46:44.290 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.290+0000 7fd2fca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:44.333 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.332+0000 7fd2fca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:44.333 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.333+0000 7fd2fca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:44.334 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.333+0000 7fd2fca15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:46:44.334 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.333+0000 7fd2fca15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:46:44.334 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.333+0000 7fd2fca15640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:46:44.408 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:44.407+0000 7fd2fca15640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:46:50.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:46:50.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:46:50.513 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:46:50.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:46:50.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:46:50.860 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:46:50.861 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:46:50.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:46:50.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:46:50.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:46:50.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:46:50.889 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:50.888+0000 7fa5f740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:50.891 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:50.890+0000 7fa5f740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:50.892 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:50.891+0000 7fa5f740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:46:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:46:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:46:51.135 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:46:51.136 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:51.137 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:51.415 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:51.415+0000 7fa5f740d640 -1 Falling back to public interface 2024-03-09T16:46:51.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:52.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:52.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:52.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:46:52.435 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:46:52.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:52.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:52.717 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:53.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:53.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:53.718 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:46:53.719 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:46:53.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:53.719 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:53.852 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:46:53.852+0000 7fa5f740d640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:46:54.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:55.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:55.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:55.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:46:55.088 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:46:55.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:55.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:55.371 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:56.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:56.373 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:46:56.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:46:56.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:56.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:56.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:57.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:57.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:57.665 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:46:57.665 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:46:57.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:57.666 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:58.043 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:46:59.045 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:46:59.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:46:59.046 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:46:59.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:46:59.046 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:46:59.047 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:46:59.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:00.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:00.327 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:00.327 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:47:00.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:47:00.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:00.328 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:00.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:01.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:01.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:01.708 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:47:01.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:47:01.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:01.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:02.077 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:03.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:03.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:03.078 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:47:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:47:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:03.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:03.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:04.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:04.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:04.361 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:47:04.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:47:04.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:04.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:04.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:05.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:05.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:05.678 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:47:05.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:47:05.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:05.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:05.995 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:06.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:06.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:06.997 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:47:06.997 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:47:06.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:06.998 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:07.578 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:08.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:08.579 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:08.579 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:47:08.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:47:08.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:08.580 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:09.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:10.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:10.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:10.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:47:10.224 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:47:10.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:10.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:10.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:11.654 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:11.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:11.655 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:47:11.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:47:11.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:11.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:12.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:13.072 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:13.072 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:47:13.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:13.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:47:13.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:13.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:13.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:14.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:14.521 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:47:14.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:14.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:47:14.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:14.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:15.177 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:16.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:16.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:16.179 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:47:16.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:47:16.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:16.179 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:16.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:17.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:17.502 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:47:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:47:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:17.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:47:17.855 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/263017047,v1:127.0.0.1:6811/263017047] [v2:127.0.0.1:6812/263017047,v1:127.0.0.1:6813/263017047] exists,up 05e24bfa-8d87-49eb-b984-98d3ebdf35c2 2024-03-09T16:47:17.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:47:17.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:47:17.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:47:17.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-03-09T16:47:17.856 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T16:47:18.117 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' does not exist 2024-03-09T16:47:18.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T16:47:18.129 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T16:47:18.521 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'rbd' created 2024-03-09T16:47:18.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:47:19.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T16:47:21.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-09T16:47:21.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:47:21.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:47:21.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:47:21.323 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:47:21.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:47:21.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:47:21.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:47:21.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:47:21.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:47:21.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:47:21.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:47:21.696 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:47:21.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:47:21.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:21.696 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:47:21.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-09T16:47:21.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-09T16:47:21.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-09T16:47:21.824 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:21.824 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:47:21.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T16:47:21.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T16:47:21.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672966' 2024-03-09T16:47:21.926 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:21.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-09T16:47:21.927 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:21.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:47:21.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-09T16:47:21.929 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:21.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-09T16:47:21.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-09T16:47:21.930 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836491 2024-03-09T16:47:21.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:22.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-03-09T16:47:22.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:47:23.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:47:23.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:23.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-09T16:47:23.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:23.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T16:47:23.485 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:23.485 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:47:23.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T16:47:23.486 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:23.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T16:47:23.487 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T16:47:23.488 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672966 2024-03-09T16:47:23.488 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:47:23.759 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-03-09T16:47:23.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:47:23.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:23.759 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:24.099 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:47:24.100 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:47:24.369 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T16:47:24.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:47:24.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:24.370 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:24.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T16:47:24.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:47:24.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:47:24.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-03-09T16:47:24.705 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-03-09T16:47:25.073 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'psr_pool' created 2024-03-09T16:47:25.084 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:47:26.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-09T16:47:26.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:47:26.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:47:26.086 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:47:26.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:47:26.189 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:47:26.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:47:26.484 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:47:26.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:47:26.484 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:26.484 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:47:26.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-09T16:47:26.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-09T16:47:26.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-09T16:47:26.588 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:26.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672968' 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-09T16:47:26.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:26.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:47:26.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-09T16:47:26.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:26.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-09T16:47:26.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-09T16:47:26.755 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836493 2024-03-09T16:47:26.755 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:27.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-09T16:47:27.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:47:28.024 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:47:28.024 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:28.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-09T16:47:28.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:28.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-09T16:47:28.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:28.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:47:28.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-09T16:47:28.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:28.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-09T16:47:28.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-09T16:47:28.303 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672968 2024-03-09T16:47:28.303 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:47:28.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-09T16:47:28.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:47:28.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:28.576 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:28.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:47:28.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:47:28.913 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:47:29.181 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:47:29.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:47:29.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:29.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:29.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:47:29.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-03-09T16:47:29.541 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-03-09T16:47:29.918 INFO:tasks.workunit.client.0.smithi146.stderr:noscrub is unset 2024-03-09T16:47:29.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-03-09T16:47:30.436 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is unset 2024-03-09T16:47:30.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T16:47:30.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T16:47:30.448 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-03-09T16:47:30.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-03-09T16:47:30.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-03-09T16:47:30.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-03-09T16:47:30.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-03-09T16:47:30.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:47:30.962 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-03-09T16:47:30.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-03-09T16:47:30.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-03-09T16:47:30.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-03-09T16:47:30.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-03-09T16:47:30.963 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:47:31.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-03-09T16:47:31.242 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:47:31.243 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:47:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:47:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:47:31.244 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:47:31.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:47:31.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:47:31.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:47:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:47:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:47:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:47:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:47:31.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:47:33.379 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:47:33.380 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:start osd.0 2024-03-09T16:47:33.381 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T16:47:33.382 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:47:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:47:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:47:33.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:47:33.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:47:33.410 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:47:33.408+0000 7f1139001640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:47:33.414 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:47:33.410+0000 7f1139001640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:47:33.414 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:47:33.411+0000 7f1139001640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:47:33.658 INFO:tasks.workunit.client.0.smithi146.stderr:0 2024-03-09T16:47:33.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:33.659 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:47:33.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:33.935 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:47:33.934+0000 7f1139001640 -1 Falling back to public interface 2024-03-09T16:47:34.929 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:34.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:34.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:47:34.930 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:47:34.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:34.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:47:35.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:36.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:36.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:36.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:47:36.211 INFO:tasks.workunit.client.0.smithi146.stderr:2 2024-03-09T16:47:36.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:36.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:47:36.282 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:47:36.281+0000 7f1139001640 -1 osd.0 25 log_to_monitors true 2024-03-09T16:47:36.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:47:37.494 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:47:37.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:47:37.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:47:37.495 INFO:tasks.workunit.client.0.smithi146.stderr:3 2024-03-09T16:47:37.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:47:37.495 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/1733015676,v1:127.0.0.1:6803/1733015676] [v2:127.0.0.1:6804/1733015676,v1:127.0.0.1:6805/1733015676] exists,up 2d8d89ce-6fdd-4a25-acdd-b44fda1a2e28 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:47:37.855 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:47:37.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:47:37.856 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:47:37.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:47:37.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:47:37.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:47:37.945 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:47:37.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:47:37.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:47:37.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:47:37.946 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:47:37.946 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:47:38.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:47:38.213 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:47:38.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:47:38.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:38.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:47:38.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T16:47:38.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T16:47:38.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-09T16:47:38.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:38.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:47:38.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-09T16:47:38.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-09T16:47:38.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672972' 2024-03-09T16:47:38.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:38.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-09T16:47:38.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:38.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:47:38.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-09T16:47:38.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:38.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T16:47:38.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-09T16:47:38.425 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.0 seq 124554051586 2024-03-09T16:47:38.425 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:38.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-09T16:47:38.697 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:47:39.696 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:47:39.697 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:39.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T16:47:39.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:39.975 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-09T16:47:39.975 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:39.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:47:39.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-09T16:47:39.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:39.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-09T16:47:39.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-09T16:47:39.978 INFO:tasks.workunit.client.0.smithi146.stderr:waiting osd.1 seq 42949672972 2024-03-09T16:47:39.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:47:40.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-09T16:47:40.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:47:40.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:40.250 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:40.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T16:47:40.587 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:47:40.588 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:47:40.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T16:47:40.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:47:40.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:47:40.859 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:47:41.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T16:47:41.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:47:41.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:47:41.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-03-09T16:47:41.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T16:47:41.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-03-09T16:47:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:No scrub information available for pg 2.0 2024-03-09T16:47:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:error 2: (2) No such file or directory 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:41.222 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:41.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:41.223 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:41.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:41.545 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-03-09T16:47:41.800 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-03-09T16:47:25.021841+0000 last_deep_scrub_stamp 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:47:41.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:41.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:41.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:41.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:41.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:41.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:42.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:25.021841+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:42.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:43.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:43.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:43.081 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:43.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:43.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:43.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:43.082 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:43.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:25.021841+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:43.365 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:44.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:44.366 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:44.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:44.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:44.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:44.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:44.367 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:44.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:25.021841+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:44.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:45.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:45.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:45.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:45.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:45.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:45.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:45.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:45.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:25.021841+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:45.921 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:46.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:46.922 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:46.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:46.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:46.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:46.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:46.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:47.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:25.021841+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:47.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:48.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:48.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:48.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T16:47:48.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:48.201 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T16:47:48.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:48.202 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T16:47:48.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:42.281027+0000 '>' 2024-03-09T16:47:25.021841+0000 2024-03-09T16:47:48.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:47:48.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T16:47:48.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-03-09T16:47:48.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T16:47:48.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T16:47:48.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:47:48.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:47:48.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:47:48.769 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:47:48.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:47:48.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:48.769 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:47:48.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-09T16:47:48.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-09T16:47:48.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-09T16:47:48.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:48.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:47:48.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T16:47:48.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T16:47:48.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672975' 2024-03-09T16:47:48.982 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:48.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-09T16:47:48.983 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:48.984 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:47:48.984 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-09T16:47:48.985 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:48.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-09T16:47:48.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-09T16:47:48.986 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051589 2024-03-09T16:47:48.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:49.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-09T16:47:49.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:47:50.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:47:50.258 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:50.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-09T16:47:50.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:50.533 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T16:47:50.533 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:50.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:47:50.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T16:47:50.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:50.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T16:47:50.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T16:47:50.536 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672975 2024-03-09T16:47:50.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:47:50.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T16:47:50.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-03-09T16:47:50.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:50.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:47:50.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:50.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:47:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:51.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-09T16:47:51.172 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:47:51.172 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": false, 2024-03-09T16:47:51.172 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T16:47:51.172 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-03-02T16:46:11.172353+0000" 2024-03-09T16:47:51.172 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:51.182 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:47:51.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:42.281027+0000 '>' 2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:51.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:52.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:52.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:52.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:47:52.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:52.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:47:52.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:52.506 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:47:52.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:42.281027+0000 '>' 2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:52.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:47:53.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:47:53.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:47:53.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:47:53.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:47:53.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:47:53.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:47:53.785 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:47:54.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:47:51.236601+0000 '>' 2024-03-09T16:47:42.281027+0000 2024-03-09T16:47:54.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:47:54.063 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-03-09T16:47:54.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T16:47:54.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-03-09T16:47:54.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T16:47:54.091 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-03-09T16:47:54.547 INFO:tasks.workunit.client.0.smithi146.stderr:nodeep-scrub is set 2024-03-09T16:47:54.558 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-03-09T16:47:54.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T16:47:54.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:47:54.662 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:47:54.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:47:54.936 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:47:54.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:47:54.936 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:54.937 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:47:55.052 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-03-09T16:47:55.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-03-09T16:47:55.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-03-09T16:47:55.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:47:55.053 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:47:55.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T16:47:55.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T16:47:55.155 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-42949672977' 2024-03-09T16:47:55.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:55.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-03-09T16:47:55.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:55.157 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:47:55.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-03-09T16:47:55.158 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:55.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-03-09T16:47:55.159 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-03-09T16:47:55.159 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051591 2024-03-09T16:47:55.159 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:47:55.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-03-09T16:47:55.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:47:55.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T16:47:55.431 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:47:55.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:47:55.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T16:47:55.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:47:55.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T16:47:55.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T16:47:55.434 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672977 2024-03-09T16:47:55.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:47:55.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T16:47:55.737 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-03-09T16:48:00.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-09T16:48:00.846 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:48:00.846 INFO:tasks.workunit.client.0.smithi146.stdout: "deep": false, 2024-03-09T16:48:00.846 INFO:tasks.workunit.client.0.smithi146.stdout: "must": false, 2024-03-09T16:48:00.846 INFO:tasks.workunit.client.0.smithi146.stdout: "stamp": "2024-03-02T16:46:20.846288+0000" 2024-03-09T16:48:00.846 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:48:00.855 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-03-09T16:48:00.856 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-03-09T16:48:00.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-03-09T16:48:00.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-03-09T16:48:01.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-03-09T16:48:01.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T16:48:01.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T16:48:01.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:48:01.894 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:48:02.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:48:02.172 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:48:02.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:48:02.172 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:48:02.173 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:48:02.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-03-09T16:48:02.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-03-09T16:48:02.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-03-09T16:48:02.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:48:02.355 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:48:02.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-09T16:48:02.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-09T16:48:02.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-42949672979' 2024-03-09T16:48:02.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:48:02.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-03-09T16:48:02.460 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:48:02.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:48:02.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-03-09T16:48:02.462 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:48:02.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-03-09T16:48:02.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-03-09T16:48:02.464 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051594 2024-03-09T16:48:02.464 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:48:02.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051594 2024-03-09T16:48:02.736 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:48:03.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:48:03.738 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:48:04.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-03-09T16:48:04.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:48:04.028 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-09T16:48:04.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-09T16:48:04.030 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672979 2024-03-09T16:48:04.030 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:48:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-09T16:48:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-03-09T16:48:04.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-03-09T16:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-03-09T16:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:48:04.298 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:48:04.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:04.567 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-03-09T16:48:04.823 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 2.0 on osd.0 to scrub 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:48:04.836 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:48:05.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T16:46:20.846288+0000 '>' 2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:05.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:48:06.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:48:06.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:48:06.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:48:06.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:48:06.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:48:06.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:48:06.104 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:48:06.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T16:46:20.846288+0000 '>' 2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:06.382 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:48:07.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:48:07.383 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:48:07.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T16:48:07.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T16:48:07.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:48:07.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:48:07.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T16:48:07.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:48:05.276742+0000 '>' 2024-03-02T16:46:20.846288+0000 2024-03-09T16:48:07.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:48:07.670 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-03-09T16:48:07.672 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T16:48:07.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-03-09T16:48:07.673 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-03-09T16:48:07.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:48:07.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:48:07.699 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:48:07.700 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:48:07.807 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:48:07.807 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:48:07.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:48:07.808 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:48:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:48:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:48:07.810 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:48:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:48:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:48:07.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:48:07.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:48:07.812 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:48:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:48:07.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:48:07.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:48:07.825 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:07.826 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:07.826 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:48:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:48:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:48:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:48:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:48:07.827 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:48:07.828 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:48:07.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:48:07.829 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:48:07.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:48:07.831 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:48:07.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:48:07.832 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:48:07.832 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:48:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:48:07.833 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:48:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:48:07.834 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:48:07.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:48:07.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:48:07.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:48:07.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:48:07.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:07.837 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:07.837 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:48:07.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:48:07.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:48:07.838 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:48:07.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:48:07.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:07.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:07.840 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-03-09T16:48:07.842 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:48:07.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:08.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:08.214 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:48:08.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:48:08.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:48:08.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:48:08.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:48:08.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:48:08.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:48:08.409 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:48:08.409 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:48:08.409 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:48:08.409 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:48:08.410 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:08.410 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:08.410 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:48:08.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:48:08.410 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:08.491 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:08.492 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:48:08.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:48:08.492 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:48:08.571 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:09.383 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:09.384 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:48:09.384 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:48:09.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:48:09.413 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-03-09T16:48:09.414 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-03-09T16:48:09.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:48:09.419 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:48:09.420 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:48:09.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:09.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:48:09.423 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:48:09.424 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:48:09.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cdbb1054-d334-4e7c-af7c-211d161cdc69 2024-03-09T16:48:09.426 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cdbb1054-d334-4e7c-af7c-211d161cdc69' 2024-03-09T16:48:09.426 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 cdbb1054-d334-4e7c-af7c-211d161cdc69 2024-03-09T16:48:09.428 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:48:09.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJkuxlcy0cGhAAXnjxmQgxHzLArok1fUNs6Q== 2024-03-09T16:48:09.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJkuxlcy0cGhAAXnjxmQgxHzLArok1fUNs6Q=="}' 2024-03-09T16:48:09.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cdbb1054-d334-4e7c-af7c-211d161cdc69 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:48:09.646 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:48:09.656 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:48:09.657 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDJkuxlcy0cGhAAXnjxmQgxHzLArok1fUNs6Q== --osd-uuid cdbb1054-d334-4e7c-af7c-211d161cdc69 2024-03-09T16:48:09.687 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.686+0000 7f263d3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:09.697 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.697+0000 7f263d3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:09.698 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.698+0000 7f263d3dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:09.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.698+0000 7f263d3dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:48:09.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.698+0000 7f263d3dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:48:09.699 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.698+0000 7f263d3dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:48:09.789 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:09.788+0000 7f263d3dd640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:48:15.517 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:48:15.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:48:15.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:48:15.518 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:48:15.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:48:15.869 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:48:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:48:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:48:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:48:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:48:15.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:48:15.897 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:15.897+0000 7fc143c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:15.899 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:15.899+0000 7fc143c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:15.900 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:15.900+0000 7fc143c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:16.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:48:16.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:48:16.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:48:16.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:48:16.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:48:16.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:16.144 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:48:16.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:48:16.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:16.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:16.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:16.420 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:16.420+0000 7fc143c87640 -1 Falling back to public interface 2024-03-09T16:48:17.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:17.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:17.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:48:17.414 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:48:17.414 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:17.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:17.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:18.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:18.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:18.694 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:48:18.694 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:48:18.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:18.695 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:18.760 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:18.760+0000 7fc143c87640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:48:19.026 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:20.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:20.028 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:20.029 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:48:20.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:48:20.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:20.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:20.339 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:21.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:21.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:21.340 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:48:21.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:48:21.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:21.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:21.679 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:22.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:22.680 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:22.681 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:48:22.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:48:22.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:22.681 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:22.961 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:23.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:23.963 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:23.963 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:48:23.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:48:23.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:23.964 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:24.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:25.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:25.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:25.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:48:25.249 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:48:25.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:25.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:25.532 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:26.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:26.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:26.534 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:48:26.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:48:26.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:26.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:26.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:27.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:27.818 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:48:27.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:48:27.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:27.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:28.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:29.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:29.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:29.102 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:48:29.102 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:48:29.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:29.103 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:29.500 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:30.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:30.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:30.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:48:30.502 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:48:30.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:30.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:31.066 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:32.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:32.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:32.068 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:48:32.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:48:32.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:32.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:32.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:33.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:33.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:33.676 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:48:33.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:48:33.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:33.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:34.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:35.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:35.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:35.318 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:48:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:48:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:35.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:36.002 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:37.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:37.003 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:37.004 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:48:37.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:48:37.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:37.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:37.563 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:38.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:38.564 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:38.565 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:48:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:48:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:38.565 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:38.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:39.940 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:39.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:39.941 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:48:39.941 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:48:39.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:39.942 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:40.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:41.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:41.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:41.546 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:48:41.547 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:48:41.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:41.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:48:41.895 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1888290365,v1:127.0.0.1:6803/1888290365] [v2:127.0.0.1:6804/1888290365,v1:127.0.0.1:6805/1888290365] exists,up cdbb1054-d334-4e7c-af7c-211d161cdc69 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:48:41.900 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:48:41.901 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:48:41.902 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ec05200e-efcd-440f-bfb7-9affdeae1965 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ec05200e-efcd-440f-bfb7-9affdeae1965' 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 ec05200e-efcd-440f-bfb7-9affdeae1965 2024-03-09T16:48:41.903 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:48:41.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpkuxlhKCnNhAA9pmopwvcI7C/91XuaZoDyw== 2024-03-09T16:48:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpkuxlhKCnNhAA9pmopwvcI7C/91XuaZoDyw=="}' 2024-03-09T16:48:41.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ec05200e-efcd-440f-bfb7-9affdeae1965 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:48:42.275 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:48:42.286 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:48:42.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDpkuxlhKCnNhAA9pmopwvcI7C/91XuaZoDyw== --osd-uuid ec05200e-efcd-440f-bfb7-9affdeae1965 2024-03-09T16:48:42.317 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.315+0000 7fe7b5610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:42.360 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.358+0000 7fe7b5610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:42.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.359+0000 7fe7b5610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:42.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.360+0000 7fe7b5610640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:48:42.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.360+0000 7fe7b5610640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:48:42.361 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.360+0000 7fe7b5610640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:48:42.431 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:42.429+0000 7fe7b5610640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:48:48.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:48:48.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:48:48.289 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:48:48.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:48:48.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:48:48.640 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:48:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:48:48.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:48:48.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:48:48.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:48:48.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:48:48.667 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:48.665+0000 7ff065bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:48.669 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:48.667+0000 7ff065bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:48.670 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:48.668+0000 7ff065bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:48:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:48.914 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:48:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:48:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:49.185 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:49.193 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:49.192+0000 7ff065bfa640 -1 Falling back to public interface 2024-03-09T16:48:50.186 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:50.187 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:50.188 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:48:50.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:48:50.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:50.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:50.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:51.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:51.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:51.463 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:48:51.463 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:48:51.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:51.464 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:51.604 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:48:51.602+0000 7ff065bfa640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:48:51.771 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:52.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:52.773 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:48:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:48:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:52.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:54.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:54.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:54.058 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:48:54.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:48:54.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:54.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:54.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:55.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:55.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:55.460 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:48:55.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:48:55.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:55.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:55.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:56.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:56.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:56.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:48:56.782 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:48:56.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:56.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:57.057 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:58.058 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:58.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:58.059 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:48:58.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:48:58.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:58.059 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:58.341 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:48:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:48:59.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:48:59.343 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:48:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:48:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:48:59.344 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:48:59.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:00.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:00.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:00.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:49:00.638 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:49:00.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:00.639 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:01.007 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:02.009 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:49:02.009 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:49:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:02.010 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:02.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:03.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:03.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:03.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:49:03.298 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:49:03.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:03.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:03.641 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:04.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:04.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:04.642 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:49:04.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:49:04.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:04.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:05.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:06.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:06.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:06.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:49:06.170 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:49:06.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:06.171 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:06.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:07.809 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:49:07.809 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:49:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:07.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:08.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:09.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:09.568 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:49:09.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:49:09.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:09.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:10.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:11.169 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:11.170 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:49:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:49:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:11.170 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:11.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:12.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:12.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:12.809 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:49:12.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:49:12.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:12.810 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:13.547 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:14.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:14.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:14.548 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:49:14.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:49:14.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:14.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:14.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:15.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:15.887 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:49:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:49:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:15.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:16.179 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3603508613,v1:127.0.0.1:6811/3603508613] [v2:127.0.0.1:6812/3603508613,v1:127.0.0.1:6813/3603508613] exists,up ec05200e-efcd-440f-bfb7-9affdeae1965 2024-03-09T16:49:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:49:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:49:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:49:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-03-09T16:49:16.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T16:49:16.617 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T16:49:16.629 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:49:17.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-03-09T16:49:18.207 INFO:tasks.workunit.client.0.smithi146.stderr:set pool 1 size to 2 2024-03-09T16:49:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-03-09T16:49:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:49:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:49:18.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:49:18.219 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:49:18.220 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:49:18.220 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:49:18.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:49:18.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:49:18.220 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:49:18.312 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:49:18.612 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:49:18.612 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:49:18.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:49:18.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:49:18.613 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:49:18.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-09T16:49:18.772 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-09T16:49:18.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-09T16:49:18.773 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:49:18.773 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:49:18.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-09T16:49:18.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-09T16:49:18.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-03-09T16:49:18.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:49:18.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-09T16:49:18.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:49:18.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:49:18.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-09T16:49:18.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:49:18.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-09T16:49:18.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-09T16:49:18.879 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836495 2024-03-09T16:49:18.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:49:19.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-09T16:49:19.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:49:20.151 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:49:20.152 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:49:20.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-09T16:49:20.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:49:20.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-09T16:49:20.434 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:49:20.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:49:20.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-09T16:49:20.435 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:49:20.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-09T16:49:20.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-09T16:49:20.437 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672965 2024-03-09T16:49:20.438 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:49:20.709 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-09T16:49:20.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:49:20.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:49:20.710 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:49:21.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:49:21.048 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:49:21.049 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:49:21.316 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T16:49:21.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:49:21.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:49:21.317 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:49:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T16:49:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T16:49:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:49:21.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T16:49:21.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:49:21.653 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:49:21.754 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:49:22.021 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:49:22.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:49:22.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:49:22.022 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-03-09T16:49:22.359 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-03-09T16:49:22.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:22.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:22.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:22.492 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:22.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:22.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:22.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:22.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:22.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:22.928 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.070 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.071 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.321 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.322 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.439 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.573 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.698 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:23.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:23.957 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.083 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.317 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.318 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.560 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.829 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.830 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:24.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:24.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.432 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.691 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.817 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:25.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:25.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:26.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T16:49:26.068 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T16:49:26.186 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T16:49:26.187 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T16:49:26.187 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:49:26.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T16:49:26.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T16:49:26.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T16:49:26.765 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-03-09T16:49:26.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-03-09T16:49:26.765 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T16:49:26.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T16:49:26.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T16:49:26.766 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:49:27.039 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-03-09T16:49:27.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-03-09T16:49:27.041 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T16:49:27.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T16:49:27.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T16:49:27.042 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:49:27.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-03-09T16:49:27.314 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:49:27.314 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:49:27.315 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:49:27.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:49:27.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:49:27.315 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:49:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:49:27.621 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:49:27.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:49:27.622 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:49:27.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:49:27.622 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:49:27.623 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:49:27.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:49:27.930 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-03-09T16:49:27.930 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:27.931 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-03-09T16:49:27.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:27.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-09T16:49:29.093 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T16:49:29.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:29.607 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:29.609 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:29.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:29.610 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T16:49:30.789 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T16:49:31.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:31.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:31.306 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-09T16:49:32.049 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:8dd16f86:::obj3:head# 2024-03-09T16:49:32.566 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:32.567 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-03-09T16:49:32.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:32.568 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:32.569 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-09T16:49:33.335 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:0ee9ae15:::obj4:head# 2024-03-09T16:49:33.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:33.861 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:33.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-09T16:49:34.680 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:c52c9666:::obj5:head# 2024-03-09T16:49:35.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:35.203 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:35.205 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-09T16:49:35.939 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:233a42c1:::obj6:head# 2024-03-09T16:49:36.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:36.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-03-09T16:49:36.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:36.460 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:36.461 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-09T16:49:37.133 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:9a09113a:::obj7:head# 2024-03-09T16:49:37.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:37.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:37.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-09T16:49:38.293 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:a557efb1:::obj8:head# 2024-03-09T16:49:38.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:38.811 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:38.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-09T16:49:39.454 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:fe351e27:::obj9:head# 2024-03-09T16:49:39.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:39.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:39.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-09T16:49:40.705 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:d1337354:::obj10:head# 2024-03-09T16:49:41.220 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:41.221 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:41.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-09T16:49:41.932 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:eeae2a94:::obj11:head# 2024-03-09T16:49:42.701 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:42.701 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:42.703 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:42.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:42.704 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-09T16:49:43.352 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:af5f95cb:::obj12:head# 2024-03-09T16:49:44.121 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:44.121 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-03-09T16:49:44.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:44.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-09T16:49:44.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:44.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:44.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:44.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:44.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:44.124 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-09T16:49:44.964 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:a61ad63e:::obj13:head# 2024-03-09T16:49:45.479 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:45.479 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-03-09T16:49:45.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:45.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-09T16:49:45.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:45.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:45.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:45.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:45.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:45.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-09T16:49:46.090 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:e0a44829:::obj14:head# 2024-03-09T16:49:46.604 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:46.605 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:46.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-09T16:49:47.192 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:ab946124:::obj15:head# 2024-03-09T16:49:47.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:47.711 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:47.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-09T16:49:48.294 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:461f8b5e:::obj16:head# 2024-03-09T16:49:48.808 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:48.809 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:48.811 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-09T16:49:49.454 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:24fc6e92:::obj17:head# 2024-03-09T16:49:49.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:49.969 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:49.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-09T16:49:50.681 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:b9836a99:::obj18:head# 2024-03-09T16:49:51.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:51.196 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:51.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:51.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-09T16:49:51.882 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:00b31b6c:::obj19:head# 2024-03-09T16:49:52.398 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T16:49:52.398 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-03-09T16:49:52.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T16:49:52.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-09T16:49:52.400 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:49:52.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:49:52.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:49:52.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:49:52.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:52.401 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-09T16:49:53.026 INFO:tasks.workunit.client.0.smithi146.stdout:remove #1:24b5611b:::obj20:head# 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:49:53.548 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:49:53.549 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:49:53.550 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:49:53.551 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:49:53.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:49:53.580 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:53.578+0000 7f671e3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:53.582 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:53.581+0000 7f671e3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:53.583 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:53.582+0000 7f671e3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:53.834 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:49:53.835 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:49:53.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:53.836 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:54.101 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:54.100+0000 7f671e3e1640 -1 Falling back to public interface 2024-03-09T16:49:54.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:55.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:55.110 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:55.111 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:49:55.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:49:55.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:55.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:55.388 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:56.371 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:56.369+0000 7f671e3e1640 -1 osd.1 17 log_to_monitors true 2024-03-09T16:49:56.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:56.389 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:56.390 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:49:56.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:49:56.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:56.390 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:56.690 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:57.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:57.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:57.692 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:49:57.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:49:57.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:57.693 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:49:57.970 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/1507004834,v1:127.0.0.1:6803/1507004834] [v2:127.0.0.1:6804/1507004834,v1:127.0.0.1:6805/1507004834] exists,up ec05200e-efcd-440f-bfb7-9affdeae1965 2024-03-09T16:49:57.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:49:57.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:49:57.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:49:57.974 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:49:57.975 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:49:57.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:49:57.977 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:49:57.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:49:57.978 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:49:57.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:49:57.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:49:57.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:49:57.979 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:49:57.997 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:57.995+0000 7fcd6680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:57.998 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:57.997+0000 7fcd6680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:57.999 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:57.998+0000 7fcd6680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:49:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:49:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:49:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:49:58.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:49:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:49:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:58.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:49:58.248 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:49:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:58.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:49:58.519 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:49:58.518+0000 7fcd6680f640 -1 Falling back to public interface 2024-03-09T16:49:58.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:49:59.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:49:59.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:49:59.543 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:49:59.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:49:59.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:49:59.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:49:59.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:00.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:00.818 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:00.818 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:50:00.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:50:00.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:00.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:00.913 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:00.911+0000 7fcd6680f640 -1 osd.0 16 log_to_monitors true 2024-03-09T16:50:01.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:02.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:50:02.100 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:50:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:02.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:02.410 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/3155406061,v1:127.0.0.1:6811/3155406061] [v2:127.0.0.1:6812/3155406061,v1:127.0.0.1:6813/3155406061] exists,up cdbb1054-d334-4e7c-af7c-211d161cdc69 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:50:02.412 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:50:02.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:50:02.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:50:02.413 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:50:02.502 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:50:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:50:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:50:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:50:02.839 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:02.839 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:50:02.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-03-09T16:50:02.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-03-09T16:50:02.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-03-09T16:50:02.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:02.986 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-03-09T16:50:03.092 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:03.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-03-09T16:50:03.093 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 98784247810 2024-03-09T16:50:03.093 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-03-09T16:50:03.093 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:03.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-03-09T16:50:03.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:50:04.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:50:04.362 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:04.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-03-09T16:50:04.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:04.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345923 2024-03-09T16:50:04.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:04.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:50:04.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345923 2024-03-09T16:50:04.646 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:04.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-03-09T16:50:04.647 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-03-09T16:50:04.648 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 85899345923 2024-03-09T16:50:04.648 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:50:04.923 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345923 2024-03-09T16:50:04.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:50:04.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:50:04.923 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:50:05.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:50:05.261 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:50:05.262 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:50:05.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:50:05.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:50:05.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:50:05.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:05.874 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:05.875 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:05.875 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:06.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:06.143 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T16:50:06.399 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-09T16:50:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T16:50:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:50:06.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:06.412 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:06.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:06.710 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:07.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:07.711 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:07.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:07.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:07.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:07.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:07.712 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:07.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:07.985 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:08.986 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:08.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:08.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:08.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:08.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:08.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:08.988 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:09.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:09.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:10.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:10.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:10.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:10.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:10.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:10.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:10.260 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:10.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:10.535 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:11.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:11.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:11.536 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:11.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:11.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:11.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:11.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:11.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:11.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:12.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:12.816 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:12.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:12.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:12.817 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:12.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:12.818 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:13.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:49:16.500620+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:13.096 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:50:14.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:50:14.098 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:50:14.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:50:14.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:50:14.098 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:50:14.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:50:14.099 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:50:07.388222+0000 '>' 2024-03-09T16:49:16.500620+0000 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:50:14.375 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:50:14.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:50:14.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:50:14.376 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:50:14.467 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:50:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:50:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:50:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:50:14.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:14.770 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:50:14.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247813 2024-03-09T16:50:14.872 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247813 2024-03-09T16:50:14.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247813' 2024-03-09T16:50:14.873 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:14.873 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247813 1-85899345926' 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247813 2024-03-09T16:50:14.977 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:14.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:50:14.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247813 2024-03-09T16:50:14.979 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:14.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247813 2024-03-09T16:50:14.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2024-03-09T16:50:14.981 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 98784247813 2024-03-09T16:50:14.981 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:15.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247812 -lt 98784247813 2024-03-09T16:50:15.252 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:50:16.253 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:50:16.253 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:16.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247813 2024-03-09T16:50:16.534 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:16.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345926 2024-03-09T16:50:16.535 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:16.536 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:50:16.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345926 2024-03-09T16:50:16.537 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:16.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-03-09T16:50:16.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345926' 2024-03-09T16:50:16.539 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 85899345926 2024-03-09T16:50:16.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:50:16.813 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345926 2024-03-09T16:50:16.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:50:16.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:50:16.814 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:50:17.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:50:17.150 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:50:17.151 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:50:17.418 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:50:17.419 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:50:17.420 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:50:17.421 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:50:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:50:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:50:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:50:17.769 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-03-09T16:50:18.024 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T16:50:18.024 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-09T16:50:08.351500+0000 16'30 24:122 [1,0] 1 [1,0] 1 16'30 2024-03-09T16:50:07.388222+0000 16'30 2024-03-09T16:50:07.388222+0000 0 1 periodic scrub scheduled @ 2024-03-10T16:50:07.388222+0000 20 0 2024-03-09T16:50:18.024 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T16:50:18.024 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T16:50:18.025 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T16:50:18.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-03-09T16:50:18.036 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:50:18.036 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:50:18.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:50:18.303 INFO:tasks.workunit.client.0.smithi146.stderr:1' 2024-03-09T16:50:18.303 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:50:18.304 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:18.304 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:50:18.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247815 2024-03-09T16:50:18.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247815 2024-03-09T16:50:18.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815' 2024-03-09T16:50:18.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:50:18.407 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:50:18.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-03-09T16:50:18.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-03-09T16:50:18.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815 1-85899345928' 2024-03-09T16:50:18.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:18.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247815 2024-03-09T16:50:18.511 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:18.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:50:18.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247815 2024-03-09T16:50:18.513 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:18.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247815 2024-03-09T16:50:18.514 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 98784247815 2024-03-09T16:50:18.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2024-03-09T16:50:18.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:18.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247815 2024-03-09T16:50:18.786 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:50:19.787 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:50:19.788 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:50:20.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247815 -lt 98784247815 2024-03-09T16:50:20.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:50:20.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345928 2024-03-09T16:50:20.067 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:50:20.067 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:50:20.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345928 2024-03-09T16:50:20.068 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:50:20.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-03-09T16:50:20.069 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345928' 2024-03-09T16:50:20.070 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 85899345928 2024-03-09T16:50:20.070 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:50:20.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-03-09T16:50:20.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-03-09T16:50:20.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-03-09T16:50:20.448 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes": 210, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects": 30, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_object_clones": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_object_copies": 60, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_missing_on_primary": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_missing": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_degraded": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_misplaced": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_unfound": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_dirty": 30, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_whiteouts": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_read": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_read_kb": 0, 2024-03-09T16:50:20.449 INFO:tasks.workunit.client.0.smithi146.stdout: "num_write": 30, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_write_kb": 30, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_scrub_errors": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shallow_scrub_errors": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_deep_scrub_errors": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_recovered": 20, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes_recovered": 140, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_keys_recovered": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_omap": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_hit_set_archive": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes_hit_set_archive": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_kb": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_kb": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_promote": 0, 2024-03-09T16:50:20.450 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_mode_high": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_mode_low": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_mode_some": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_mode_full": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_pinned": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_legacy_snapsets": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_large_omap_objects": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_manifest": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_omap_bytes": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_omap_keys": 0, 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_repaired": 20 2024-03-09T16:50:20.451 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:50:20.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-03-09T16:50:20.452 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T16:50:20.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-03-09T16:50:20.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-03-09T16:50:20.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:20.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-03-09T16:50:20.807 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "osd": 1, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "up_from": 20, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "seq": 85899345928, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "num_pgs": 1, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "num_osds": 1, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_osds": 1, 2024-03-09T16:50:20.818 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb": 104857600, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used": 28552, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_data": 320, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_omap": 4, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_meta": 28219, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_avail": 104829048, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "statfs": { 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "total": 107374182400, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "available": 107344945152, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "internally_reserved": 0, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "allocated": 327680, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "data_stored": 53611, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed": 0, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_allocated": 0, 2024-03-09T16:50:20.819 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_original": 0, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "omap_allocated": 4395, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "internal_metadata": 28896981 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "hb_peers": [ 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: 0 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "snap_trim_queue_len": 0, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "num_snap_trimming": 0, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shards_repaired": 10, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "op_queue_age_hist": { 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "histogram": [], 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "upper_bound": 1 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:20.820 INFO:tasks.workunit.client.0.smithi146.stdout: "perf_stat": { 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ms": 0, 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ms": 0, 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ns": 0, 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ns": 0 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout: "alerts": [] 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:50:20.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:20.822 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-03-09T16:50:21.075 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:21.087 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-03-09T16:50:21.087 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-03-09T16:50:21.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-03-09T16:50:21.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:21.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-09T16:50:21.344 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "osd": 0, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "up_from": 23, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "seq": 98784247815, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "num_pgs": 1, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "num_osds": 1, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_osds": 1, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "kb": 104857600, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used": 28552, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_data": 320, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_omap": 4, 2024-03-09T16:50:21.355 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_meta": 28219, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_avail": 104829048, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "statfs": { 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "total": 107374182400, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "available": 107344945152, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "internally_reserved": 0, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "allocated": 327680, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "data_stored": 53611, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed": 0, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_allocated": 0, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_original": 0, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "omap_allocated": 4401, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "internal_metadata": 28896975 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: "hb_peers": [ 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: 1 2024-03-09T16:50:21.356 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "snap_trim_queue_len": 0, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "num_snap_trimming": 0, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shards_repaired": 10, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "op_queue_age_hist": { 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "histogram": [], 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "upper_bound": 1 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "perf_stat": { 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ms": 0, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ms": 0, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ns": 0, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ns": 0 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.357 INFO:tasks.workunit.client.0.smithi146.stdout: "alerts": [] 2024-03-09T16:50:21.358 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:50:21.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:21.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-09T16:50:21.613 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:21.624 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-03-09T16:50:21.625 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-03-09T16:50:21.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-03-09T16:50:21.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:21.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-03-09T16:50:21.878 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:21.889 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:50:21.889 INFO:tasks.workunit.client.0.smithi146.stdout: "up_from": 0, 2024-03-09T16:50:21.889 INFO:tasks.workunit.client.0.smithi146.stdout: "seq": 0, 2024-03-09T16:50:21.889 INFO:tasks.workunit.client.0.smithi146.stdout: "num_pgs": 2, 2024-03-09T16:50:21.889 INFO:tasks.workunit.client.0.smithi146.stdout: "num_osds": 2, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_osds": 2, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_omap_osds": 2, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb": 209715200, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used": 57104, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_data": 640, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_omap": 8, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_meta": 56439, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_avail": 209658096, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "statfs": { 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "total": 214748364800, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "available": 214689890304, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "internally_reserved": 0, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "allocated": 655360, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "data_stored": 107222, 2024-03-09T16:50:21.890 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed": 0, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_allocated": 0, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_original": 0, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "omap_allocated": 8796, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "internal_metadata": 57793956 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "hb_peers": [], 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "snap_trim_queue_len": 0, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "num_snap_trimming": 0, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shards_repaired": 20, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "op_queue_age_hist": { 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "histogram": [], 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "upper_bound": 1 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "perf_stat": { 2024-03-09T16:50:21.891 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ms": 0, 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ms": 0, 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ns": 0, 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ns": 0 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: "alerts": [], 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout: "network_ping_times": [] 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T16:50:21.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-03-09T16:50:22.144 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:50:22.156 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:50:22.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:50:22.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:50:22.158 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:50:22.264 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:50:22.265 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:50:22.265 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:50:22.266 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:50:22.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:50:22.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:50:22.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:50:22.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:50:22.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:50:22.269 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:50:22.270 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:50:22.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:50:22.271 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:50:22.287 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:50:22.288 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:50:22.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:50:22.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:50:22.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:50:22.289 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:50:22.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:50:22.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:50:22.290 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:50:22.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:50:22.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:50:22.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:50:22.292 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:50:22.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:50:22.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T16:50:22.294 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:50:22.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:50:22.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.295 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.295 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:50:22.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:50:22.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:50:22.296 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T16:50:22.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T16:50:22.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69596 2024-03-09T16:50:22.300 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T16:50:22.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T16:50:22.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T16:50:22.301 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T16:50:22.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T16:50:22.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T16:50:22.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T16:50:22.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T16:50:22.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:50:22.663 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:50:22.663 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:50:22.664 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.664 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.664 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:22.664 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T16:50:22.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T16:50:22.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T16:50:22.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:50:22.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:50:22.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T16:50:22.865 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:50:22.867 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get fsid 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T16:50:22.949 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T16:50:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:22.950 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69596/ceph-mon.a.asok 2024-03-09T16:50:22.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T16:50:22.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69596/ceph-mon.a.asok config get mon_host 2024-03-09T16:50:23.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-03-09T16:50:23.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T16:50:23.029 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T16:50:23.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T16:50:23.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T16:50:23.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T16:50:23.030 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T16:50:23.841 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:23.842 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:50:23.843 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T16:50:23.868 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:50:23.869 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-03-09T16:50:23.870 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:50:23.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:50:23.877 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:50:23.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99ea6311-c283-4eca-b4f6-029d20436979 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stdout:add osd0 99ea6311-c283-4eca-b4f6-029d20436979 2024-03-09T16:50:23.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 99ea6311-c283-4eca-b4f6-029d20436979' 2024-03-09T16:50:23.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:50:23.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPk+xleg4wNRAAWdOvF5fxEOyqJLxfQnvW0Q== 2024-03-09T16:50:23.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPk+xleg4wNRAAWdOvF5fxEOyqJLxfQnvW0Q=="}' 2024-03-09T16:50:23.894 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99ea6311-c283-4eca-b4f6-029d20436979 -i td/osd-scrub-repair/0/new.json 2024-03-09T16:50:24.130 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:50:24.140 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T16:50:24.141 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBPk+xleg4wNRAAWdOvF5fxEOyqJLxfQnvW0Q== --osd-uuid 99ea6311-c283-4eca-b4f6-029d20436979 2024-03-09T16:50:24.172 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.171+0000 7f2b82bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:24.181 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.180+0000 7f2b82bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:24.182 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.181+0000 7f2b82bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:24.182 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.182+0000 7f2b82bbd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:50:24.182 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.182+0000 7f2b82bbd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:50:24.183 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.182+0000 7f2b82bbd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T16:50:24.256 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:24.255+0000 7f2b82bbd640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T16:50:29.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T16:50:29.740 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:50:29.741 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd0 key to auth repository 2024-03-09T16:50:29.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T16:50:29.741 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:50:30.096 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:50:30.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T16:50:30.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:50:30.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:50:30.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:50:30.097 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:50:30.124 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:30.123+0000 7f9374993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:30.138 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:30.136+0000 7f9374993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:30.138 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:30.138+0000 7f9374993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:30.372 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:50:30.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:50:30.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:30.373 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:30.648 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:30.659 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:30.659+0000 7f9374993640 -1 Falling back to public interface 2024-03-09T16:50:31.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:31.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:31.649 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:50:31.650 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:50:31.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:31.650 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:31.932 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:32.933 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:32.933 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:50:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:50:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:32.934 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:32.969 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:32.968+0000 7f9374993640 -1 osd.0 0 log_to_monitors true 2024-03-09T16:50:33.228 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:34.229 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:34.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:34.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:50:34.230 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:50:34.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:34.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:34.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:35.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:35.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:35.515 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:50:35.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:50:35.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:35.515 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:35.915 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:36.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:36.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:36.917 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:50:36.917 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:50:36.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:36.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:37.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:38.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:38.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:38.199 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:50:38.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:50:38.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:38.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:38.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:39.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:39.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:39.556 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:50:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:50:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:39.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:40.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:40.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:40.913 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:50:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:50:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:40.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:41.195 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:42.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:42.197 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:42.198 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:50:42.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:50:42.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:42.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:42.552 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:43.554 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:43.554 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:50:43.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:43.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:50:43.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:43.555 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:43.857 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:44.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:44.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:44.859 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:50:44.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:50:44.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:44.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:45.374 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:46.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:46.376 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:46.376 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:50:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:50:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:46.377 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:47.206 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:48.208 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:48.208 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:50:48.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:50:48.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:48.209 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:48.976 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:49.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:49.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:49.977 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:50:49.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:50:49.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:49.978 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:50.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:51.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:51.677 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:51.677 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:50:51.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:50:51.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:51.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:52.162 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:53.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:53.163 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:53.163 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:50:53.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:50:53.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:53.164 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:53.503 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:54.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:54.504 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:54.505 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:50:54.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:50:54.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:54.505 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:54.967 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:50:55.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:50:55.968 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:50:55.969 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:50:55.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:50:55.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:50:55.969 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:50:56.241 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1600781573,v1:127.0.0.1:6803/1600781573] [v2:127.0.0.1:6804/1600781573,v1:127.0.0.1:6805/1600781573] exists,up 99ea6311-c283-4eca-b4f6-029d20436979 2024-03-09T16:50:56.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:50:56.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:50:56.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:50:56.246 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:50:56.247 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43435d7c-272e-4604-8f41-fe1cdea56964 2024-03-09T16:50:56.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 43435d7c-272e-4604-8f41-fe1cdea56964' 2024-03-09T16:50:56.249 INFO:tasks.workunit.client.0.smithi146.stdout:add osd1 43435d7c-272e-4604-8f41-fe1cdea56964 2024-03-09T16:50:56.249 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:50:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwk+xlmreoDxAALmciaP0U05ASrcywUkrppg== 2024-03-09T16:50:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwk+xlmreoDxAALmciaP0U05ASrcywUkrppg=="}' 2024-03-09T16:50:56.263 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43435d7c-272e-4604-8f41-fe1cdea56964 -i td/osd-scrub-repair/1/new.json 2024-03-09T16:50:56.607 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:50:56.618 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T16:50:56.619 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBwk+xlmreoDxAALmciaP0U05ASrcywUkrppg== --osd-uuid 43435d7c-272e-4604-8f41-fe1cdea56964 2024-03-09T16:50:56.649 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.648+0000 7fb39120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:56.683 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.682+0000 7fb39120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:56.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.683+0000 7fb39120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:50:56.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.684+0000 7fb39120f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:50:56.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.684+0000 7fb39120f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:50:56.684 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.684+0000 7fb39120f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T16:50:56.749 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:50:56.749+0000 7fb39120f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T16:51:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T16:51:02.415 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:51:02.416 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T16:51:02.416 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd1 key to auth repository 2024-03-09T16:51:02.417 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:51:02.763 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:51:02.763 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T16:51:02.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:51:02.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:51:02.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:51:02.764 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:51:02.791 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:02.790+0000 7f661256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:02.792 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:02.792+0000 7f661256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:02.794 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:02.793+0000 7f661256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:03.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:51:03.076 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:51:03.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:03.076 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:03.315 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:03.315+0000 7f661256f640 -1 Falling back to public interface 2024-03-09T16:51:03.348 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:04.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:04.350 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:04.350 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:51:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:51:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:04.351 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:04.630 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:05.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:05.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:05.632 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:51:05.632 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:51:05.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:05.633 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:05.670 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:05.669+0000 7f661256f640 -1 osd.1 0 log_to_monitors true 2024-03-09T16:51:05.970 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:06.971 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:06.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:06.972 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:51:06.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:51:06.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:06.973 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:07.255 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:08.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:08.256 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:08.256 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:51:08.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:51:08.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:08.257 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:08.625 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:09.626 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:09.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:09.627 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:51:09.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:51:09.627 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:09.628 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:09.908 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:10.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:10.909 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:10.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:51:10.910 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:51:10.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:10.910 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:11.223 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:12.224 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:12.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:12.225 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:51:12.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:51:12.225 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:12.226 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:12.605 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:13.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:13.606 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:13.607 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:51:13.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:51:13.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:13.607 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:13.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:14.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:14.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:14.887 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:51:14.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:51:14.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:14.888 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:15.230 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:16.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:16.231 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:16.231 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:51:16.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:51:16.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:16.232 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:16.510 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:17.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:17.511 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:17.512 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:51:17.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:51:17.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:17.512 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:17.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:18.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:18.797 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:51:18.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:18.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:51:18.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:18.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:19.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:20.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:20.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:20.435 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:51:20.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:51:20.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:20.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:20.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:21.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:21.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:21.784 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:51:21.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:51:21.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:21.785 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:22.324 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:23.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:23.325 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:23.326 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:51:23.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:51:23.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:23.326 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:23.874 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:24.875 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:24.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:24.876 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:51:24.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:51:24.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:24.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:25.188 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:26.189 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:26.190 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:51:26.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:51:26.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:26.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:26.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:27.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:27.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:27.748 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:51:27.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:51:27.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:27.749 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:28.519 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:29.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:29.520 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:29.520 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:51:29.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:51:29.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:29.521 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:51:29.798 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2923432408,v1:127.0.0.1:6811/2923432408] [v2:127.0.0.1:6812/2923432408,v1:127.0.0.1:6813/2923432408] exists,up 43435d7c-272e-4604-8f41-fe1cdea56964 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:51:29.803 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:51:29.804 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T16:51:29.805 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1db3be8-9f36-478e-abae-42e65006ede8 2024-03-09T16:51:29.806 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e1db3be8-9f36-478e-abae-42e65006ede8' 2024-03-09T16:51:29.806 INFO:tasks.workunit.client.0.smithi146.stdout:add osd2 e1db3be8-9f36-478e-abae-42e65006ede8 2024-03-09T16:51:29.806 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T16:51:29.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRk+xlSinfMBAAkJxv5Gh+oEIfYDo7bS4S5w== 2024-03-09T16:51:29.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRk+xlSinfMBAAkJxv5Gh+oEIfYDo7bS4S5w=="}' 2024-03-09T16:51:29.820 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1db3be8-9f36-478e-abae-42e65006ede8 -i td/osd-scrub-repair/2/new.json 2024-03-09T16:51:30.237 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:51:30.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T16:51:30.249 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCRk+xlSinfMBAAkJxv5Gh+oEIfYDo7bS4S5w== --osd-uuid e1db3be8-9f36-478e-abae-42e65006ede8 2024-03-09T16:51:30.279 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.278+0000 7f7df209b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:30.335 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.334+0000 7f7df209b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:30.335 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.335+0000 7f7df209b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:30.336 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.336+0000 7f7df209b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:51:30.336 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.336+0000 7f7df209b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:51:30.336 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.336+0000 7f7df209b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T16:51:30.413 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:30.412+0000 7f7df209b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T16:51:36.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T16:51:36.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T16:51:36.364 INFO:tasks.workunit.client.0.smithi146.stdout:adding osd2 key to auth repository 2024-03-09T16:51:36.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T16:51:36.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T16:51:36.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T16:51:36.716 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.2 2024-03-09T16:51:36.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:51:36.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T16:51:36.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T16:51:36.716 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T16:51:36.743 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:36.742+0000 7fc836349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:36.744 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:36.744+0000 7fc836349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:36.745 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:36.745+0000 7fc836349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:51:37.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T16:51:37.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:51:37.015 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:51:37.016 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:37.017 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:37.267 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:37.267+0000 7fc836349640 -1 Falling back to public interface 2024-03-09T16:51:37.289 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:38.290 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:38.291 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:51:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:51:38.291 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:38.292 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:38.574 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:39.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:39.576 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:51:39.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:39.576 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:51:39.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:39.577 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:39.611 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:51:39.610+0000 7fc836349640 -1 osd.2 0 log_to_monitors true 2024-03-09T16:51:39.952 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:40.953 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:40.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:40.954 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:51:40.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:51:40.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:40.955 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:41.236 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:42.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:42.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:42.238 INFO:tasks.workunit.client.0.smithi146.stdout:4 2024-03-09T16:51:42.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T16:51:42.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:42.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:42.592 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:43.593 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:43.594 INFO:tasks.workunit.client.0.smithi146.stdout:5 2024-03-09T16:51:43.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T16:51:43.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:43.594 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:43.876 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:44.877 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:44.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:44.878 INFO:tasks.workunit.client.0.smithi146.stdout:6 2024-03-09T16:51:44.878 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T16:51:44.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:44.879 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:45.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:46.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T16:51:46.235 INFO:tasks.workunit.client.0.smithi146.stdout:7 2024-03-09T16:51:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:46.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:46.518 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:47.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:47.522 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:47.522 INFO:tasks.workunit.client.0.smithi146.stdout:8 2024-03-09T16:51:47.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T16:51:47.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:47.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:47.800 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:48.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:48.801 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:48.801 INFO:tasks.workunit.client.0.smithi146.stdout:9 2024-03-09T16:51:48.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T16:51:48.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:48.802 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:49.200 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:50.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:50.201 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:50.202 INFO:tasks.workunit.client.0.smithi146.stdout:10 2024-03-09T16:51:50.202 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T16:51:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:50.203 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:50.489 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:51.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:51.490 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:51.491 INFO:tasks.workunit.client.0.smithi146.stdout:11 2024-03-09T16:51:51.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T16:51:51.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:51.491 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:51.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:52.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:52.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:52.778 INFO:tasks.workunit.client.0.smithi146.stdout:12 2024-03-09T16:51:52.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T16:51:52.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:52.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:53.144 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:54.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:54.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:54.145 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T16:51:54.145 INFO:tasks.workunit.client.0.smithi146.stdout:13 2024-03-09T16:51:54.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:54.146 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:54.905 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:55.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:55.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:55.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T16:51:55.907 INFO:tasks.workunit.client.0.smithi146.stdout:14 2024-03-09T16:51:55.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:55.907 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:56.340 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:57.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:57.342 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:57.342 INFO:tasks.workunit.client.0.smithi146.stdout:15 2024-03-09T16:51:57.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T16:51:57.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:57.343 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:57.912 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:51:58.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:51:58.913 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:51:58.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T16:51:58.914 INFO:tasks.workunit.client.0.smithi146.stdout:16 2024-03-09T16:51:58.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:51:58.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:51:59.266 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:00.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:00.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:00.268 INFO:tasks.workunit.client.0.smithi146.stdout:17 2024-03-09T16:52:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T16:52:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:00.268 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:52:00.869 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:01.870 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:01.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:01.871 INFO:tasks.workunit.client.0.smithi146.stdout:18 2024-03-09T16:52:01.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T16:52:01.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:01.871 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:52:02.792 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:03.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:03.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:03.794 INFO:tasks.workunit.client.0.smithi146.stdout:19 2024-03-09T16:52:03.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T16:52:03.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:03.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T16:52:04.099 INFO:tasks.workunit.client.0.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1860565548,v1:127.0.0.1:6819/1860565548] [v2:127.0.0.1:6820/1860565548,v1:127.0.0.1:6821/1860565548] exists,up e1db3be8-9f36-478e-abae-42e65006ede8 2024-03-09T16:52:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:52:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:52:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:52:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-03-09T16:52:04.100 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-03-09T16:52:04.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T16:52:04.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T16:52:04.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T16:52:04.101 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T16:52:04.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-03-09T16:52:04.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-03-09T16:52:05.097 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'testpool' created 2024-03-09T16:52:05.109 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:52:06.111 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:52:06.112 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:52:06.112 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:52:06.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:52:06.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:52:06.112 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:52:06.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:52:06.211 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:52:06.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:52:06.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:52:06.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:52:06.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:52:06.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:52:06.497 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:52:06.497 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:52:06.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:52:06.498 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:52:06.499 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:52:06.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T16:52:06.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T16:52:06.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T16:52:06.601 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:52:06.601 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:52:06.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-09T16:52:06.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-09T16:52:06.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-03-09T16:52:06.715 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:52:06.715 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:52:06.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T16:52:06.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T16:52:06.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509445' 2024-03-09T16:52:06.819 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:52:06.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T16:52:06.820 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:52:06.821 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:52:06.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T16:52:06.821 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:52:06.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T16:52:06.823 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T16:52:06.823 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 21474836500 2024-03-09T16:52:06.823 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:52:07.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T16:52:07.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:52:08.176 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:52:08.176 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:52:08.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T16:52:08.455 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:52:08.455 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-09T16:52:08.456 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:52:08.456 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:52:08.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-09T16:52:08.457 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:52:08.458 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-09T16:52:08.459 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 42949672974 2024-03-09T16:52:08.459 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-09T16:52:08.459 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:52:08.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-09T16:52:08.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:52:08.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T16:52:08.729 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:52:08.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:52:08.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T16:52:08.731 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:52:08.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T16:52:08.732 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T16:52:08.732 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509445 2024-03-09T16:52:08.732 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:52:09.004 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-09T16:52:09.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:52:09.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:52:09.005 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:52:09.345 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:52:09.346 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:52:09.613 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T16:52:09.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:52:09.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:52:09.614 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:52:09.972 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T16:52:10.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:52:10.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:52:10.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:52:10.073 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:52:10.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:52:10.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:52:10.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:52:10.074 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:52:10.368 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T16:52:10.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:52:10.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:52:10.369 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:52:10.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:52:10.707 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:52:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:52:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:52:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T16:52:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:52:11.044 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T16:52:11.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:52:11.245 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:52:11.246 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:52:11.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T16:52:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:52:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:52:11.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T16:52:11.854 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T16:52:12.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T16:52:12.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T16:52:12.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T16:52:12.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T16:52:12.190 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T16:52:12.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T16:52:12.590 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:52:12.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:52:12.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:52:12.591 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:52:12.592 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:52:12.592 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:52:12.592 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:52:12.858 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:52:12.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:52:12.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:52:12.858 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-03-09T16:52:13.198 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-03-09T16:52:13.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:13.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:13.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:13.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:13.636 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:13.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:13.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:14.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:14.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:14.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:14.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:14.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:14.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:14.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:14.692 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:14.918 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:14.919 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:15.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:15.154 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:15.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:15.363 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:15.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:15.556 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:15.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:15.757 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:15.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:15.950 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:16.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:16.168 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:16.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:16.361 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:16.637 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:16.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:16.863 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:16.864 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:17.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:17.082 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:17.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:17.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:17.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:17.476 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:17.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:17.661 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:17.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:17.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:18.088 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:18.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:18.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:18.298 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:18.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:18.499 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:18.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:18.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:18.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:18.977 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:19.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:19.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:19.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T16:52:19.480 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-09T16:52:19.690 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T16:52:19.691 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:52:19.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T16:52:19.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T16:52:19.991 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T16:52:20.267 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-03-09T16:52:20.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-03-09T16:52:20.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T16:52:20.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T16:52:20.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T16:52:20.268 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T16:52:20.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-03-09T16:52:20.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-03-09T16:52:20.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T16:52:20.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T16:52:20.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T16:52:20.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T16:52:20.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-03-09T16:52:20.814 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T16:52:20.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:52:20.815 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:52:20.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:52:20.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:52:20.815 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:52:21.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:52:21.122 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T16:52:21.122 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:52:21.123 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:52:21.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:52:21.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:52:21.123 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:52:21.430 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:52:21.430 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-03-09T16:52:21.431 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:21.432 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-03-09T16:52:21.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:21.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-09T16:52:21.433 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:21.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:21.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:21.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:21.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:21.434 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-09T16:52:22.724 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-03-09T16:52:23.238 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:23.239 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:23.241 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T16:52:24.389 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:104778fc:::obj2:head# 2024-03-09T16:52:24.903 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:24.904 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:24.906 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-09T16:52:25.682 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-03-09T16:52:26.196 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:26.197 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:26.199 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-09T16:52:26.956 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-03-09T16:52:27.478 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:27.479 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:27.481 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:27.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:27.482 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-09T16:52:28.220 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:c52c9666:::obj5:head# 2024-03-09T16:52:28.735 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:28.736 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:28.738 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-09T16:52:29.590 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:233a42c1:::obj6:head# 2024-03-09T16:52:30.104 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:30.105 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:30.107 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:30.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:30.108 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-09T16:52:30.758 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:9a09113a:::obj7:head# 2024-03-09T16:52:31.273 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:31.274 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-03-09T16:52:31.275 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:31.276 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-09T16:52:31.894 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:a557efb1:::obj8:head# 2024-03-09T16:52:32.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:32.409 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-03-09T16:52:32.410 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:32.411 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-09T16:52:33.138 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:fe351e27:::obj9:head# 2024-03-09T16:52:33.652 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:33.653 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:33.655 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-09T16:52:34.364 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:d1337354:::obj10:head# 2024-03-09T16:52:34.880 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:34.880 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-03-09T16:52:34.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:34.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-09T16:52:34.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:34.883 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:34.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:34.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:34.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:34.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-09T16:52:35.525 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-03-09T16:52:36.297 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:36.297 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:36.299 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:36.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:36.300 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-09T16:52:36.962 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-03-09T16:52:37.728 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:37.728 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:37.730 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-09T16:52:38.331 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-03-09T16:52:38.846 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:38.847 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:38.849 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-09T16:52:39.433 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:e0a44829:::obj14:head# 2024-03-09T16:52:39.951 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:39.952 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:39.954 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-09T16:52:40.701 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:ab946124:::obj15:head# 2024-03-09T16:52:41.217 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:41.217 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:41.219 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-09T16:52:41.836 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-03-09T16:52:42.352 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:42.352 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-03-09T16:52:42.354 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:42.355 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-09T16:52:42.978 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-03-09T16:52:43.493 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:43.494 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:43.496 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-09T16:52:44.190 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:b9836a99:::obj18:head# 2024-03-09T16:52:44.706 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:44.706 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:44.708 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-09T16:52:45.367 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-03-09T16:52:45.881 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:45.882 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:45.884 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-09T16:52:46.493 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:24b5611b:::obj20:head# 2024-03-09T16:52:47.020 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:47.020 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-03-09T16:52:47.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:47.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-03-09T16:52:47.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:47.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:47.022 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:47.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:47.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:47.023 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-03-09T16:52:47.696 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-03-09T16:52:48.210 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:48.211 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:48.213 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-03-09T16:52:48.898 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:c0c49def:::obj22:head# 2024-03-09T16:52:49.422 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:49.423 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:49.425 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-03-09T16:52:50.125 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:242c40e0:::obj23:head# 2024-03-09T16:52:50.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:50.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-03-09T16:52:50.645 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:50.646 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-03-09T16:52:51.369 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:7eacede5:::obj24:head# 2024-03-09T16:52:51.887 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:51.888 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:51.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-03-09T16:52:52.533 INFO:tasks.workunit.client.0.smithi146.stdout:remove 0#1:7109478f:::obj25:head# 2024-03-09T16:52:53.051 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T16:52:53.051 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-03-09T16:52:53.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T16:52:53.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-03-09T16:52:53.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T16:52:53.053 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T16:52:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T16:52:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T16:52:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-03-09T16:52:53.714 INFO:tasks.workunit.client.0.smithi146.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:52:54.233 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:52:54.234 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:52:54.235 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T16:52:54.236 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.1 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:52:54.237 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:52:54.265 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:54.264+0000 7ff4ffdff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:54.267 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:54.266+0000 7ff4ffdff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:54.268 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:54.268+0000 7ff4ffdff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:52:54.523 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:52:54.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:54.524 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:52:54.787 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:54.787+0000 7ff4ffdff640 -1 Falling back to public interface 2024-03-09T16:52:54.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:55.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:55.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:55.796 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:52:55.796 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:52:55.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:55.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:52:56.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:57.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:57.080 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:57.080 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:52:57.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:52:57.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:57.081 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:52:57.118 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:57.118+0000 7ff4ffdff640 -1 osd.1 22 log_to_monitors true 2024-03-09T16:52:57.406 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:52:58.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:52:58.407 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:58.407 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:52:58.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:52:58.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:58.408 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T16:52:58.680 INFO:tasks.workunit.client.0.smithi146.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/3551977780,v1:127.0.0.1:6803/3551977780] [v2:127.0.0.1:6804/3551977780,v1:127.0.0.1:6805/3551977780] exists,up 43435d7c-272e-4604-8f41-fe1cdea56964 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T16:52:58.684 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T16:52:58.685 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T16:52:58.686 INFO:tasks.workunit.client.0.smithi146.stdout:start osd.0 2024-03-09T16:52:58.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=76463706-5f32-4a78-a76f-5021b54a6384 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69596/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T16:52:58.687 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T16:52:58.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T16:52:58.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T16:52:58.687 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T16:52:58.688 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T16:52:58.710 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:58.710+0000 7f430491d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:58.712 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:58.712+0000 7f430491d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:58.713 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:58.713+0000 7f430491d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T16:52:58.987 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:52:58.988 INFO:tasks.workunit.client.0.smithi146.stdout:0 2024-03-09T16:52:58.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T16:52:58.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:52:58.988 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:52:59.234 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:52:59.234+0000 7f430491d640 -1 Falling back to public interface 2024-03-09T16:52:59.258 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:53:00.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:53:00.259 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:53:00.260 INFO:tasks.workunit.client.0.smithi146.stdout:1 2024-03-09T16:53:00.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T16:53:00.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:53:00.260 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:53:00.542 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:53:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:53:01.543 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:53:01.544 INFO:tasks.workunit.client.0.smithi146.stdout:2 2024-03-09T16:53:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T16:53:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:53:01.544 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:53:01.555 INFO:tasks.workunit.client.0.smithi146.stderr:2024-03-09T16:53:01.555+0000 7f430491d640 -1 osd.0 21 log_to_monitors true 2024-03-09T16:53:01.859 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T16:53:02.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T16:53:02.861 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T16:53:02.861 INFO:tasks.workunit.client.0.smithi146.stdout:3 2024-03-09T16:53:02.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T16:53:02.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T16:53:02.862 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T16:53:03.210 INFO:tasks.workunit.client.0.smithi146.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/2667418589,v1:127.0.0.1:6811/2667418589] [v2:127.0.0.1:6812/2667418589,v1:127.0.0.1:6813/2667418589] exists,up 99ea6311-c283-4eca-b4f6-029d20436979 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:53:03.212 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:53:03.213 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:53:03.213 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:53:03.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:53:03.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:53:03.213 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:53:03.302 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:03.575 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:53:03.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T16:53:03.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T16:53:03.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-09T16:53:03.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:03.678 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:53:03.781 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-03-09T16:53:03.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-03-09T16:53:03.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-03-09T16:53:03.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:03.782 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:53:03.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-09T16:53:03.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-09T16:53:03.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509457' 2024-03-09T16:53:03.885 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:03.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-09T16:53:03.886 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:03.890 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:53:03.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-09T16:53:03.891 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:03.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T16:53:03.891 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-09T16:53:03.891 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051586 2024-03-09T16:53:03.892 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:53:04.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T16:53:04.178 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:04.179 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-03-09T16:53:04.179 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:04.180 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:53:04.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-03-09T16:53:04.181 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:04.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-03-09T16:53:04.182 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-03-09T16:53:04.183 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 111669149699 2024-03-09T16:53:04.183 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:53:04.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-03-09T16:53:04.470 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:04.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-09T16:53:04.471 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:04.472 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:53:04.472 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-09T16:53:04.473 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:04.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-09T16:53:04.474 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-09T16:53:04.474 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509457 2024-03-09T16:53:04.474 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:53:04.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-09T16:53:04.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:53:04.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:53:04.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:53:05.089 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:53:05.090 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:53:05.090 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:53:05.367 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:53:05.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:53:05.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:53:05.368 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:53:05.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:53:05.712 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:53:05.713 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:53:05.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:05.980 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T16:53:06.235 INFO:tasks.workunit.client.0.smithi146.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:53:06.248 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:53:06.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:52:04.987780+0000 '>' 2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:06.513 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:53:07.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:53:07.514 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:53:07.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:53:07.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:53:07.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:53:07.515 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:53:07.516 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:53:07.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:52:04.987780+0000 '>' 2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:07.797 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T16:53:08.798 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T16:53:08.799 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T16:53:08.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T16:53:08.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T16:53:08.799 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T16:53:08.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T16:53:08.800 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T16:53:07.157671+0000 '>' 2024-03-09T16:52:04.987780+0000 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T16:53:09.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:53:09.166 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:53:09.435 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:53:09.435 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:53:09.436 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:53:09.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:53:09.436 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:09.436 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:53:09.538 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051588 2024-03-09T16:53:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051588 2024-03-09T16:53:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051588' 2024-03-09T16:53:09.539 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:09.539 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:53:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149701 2024-03-09T16:53:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149701 2024-03-09T16:53:09.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051588 1-111669149701' 2024-03-09T16:53:09.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:09.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:53:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-09T16:53:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-09T16:53:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051588 1-111669149701 2-64424509460' 2024-03-09T16:53:09.747 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:09.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051588 2024-03-09T16:53:09.748 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:09.748 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:53:09.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051588 2024-03-09T16:53:09.749 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:09.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051588 2024-03-09T16:53:09.751 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051588 2024-03-09T16:53:09.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051588' 2024-03-09T16:53:09.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:53:10.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051587 -lt 124554051588 2024-03-09T16:53:10.073 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:53:11.074 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:53:11.075 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:53:11.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051588 2024-03-09T16:53:11.356 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:11.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149701 2024-03-09T16:53:11.356 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:11.357 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:53:11.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149701 2024-03-09T16:53:11.358 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149701 2024-03-09T16:53:11.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149701' 2024-03-09T16:53:11.360 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 111669149701 2024-03-09T16:53:11.360 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:53:11.674 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149701 2024-03-09T16:53:11.675 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:11.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-09T16:53:11.675 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:11.676 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:53:11.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-09T16:53:11.677 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-09T16:53:11.678 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509460 2024-03-09T16:53:11.678 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-09T16:53:11.679 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:53:11.949 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-09T16:53:11.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T16:53:11.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:53:11.950 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T16:53:12.293 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T16:53:12.561 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T16:53:12.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T16:53:12.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T16:53:12.561 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T16:53:12.897 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T16:53:12.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T16:53:12.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T16:53:12.898 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-03-09T16:53:13.156 INFO:tasks.workunit.client.0.smithi146.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T16:53:13.156 INFO:tasks.workunit.client.0.smithi146.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-09T16:53:08.560536+0000 21'30 30:130 [1,0,2] 1 [1,0,2] 1 21'30 2024-03-09T16:53:07.157671+0000 21'30 2024-03-09T16:53:07.157671+0000 0 1 periodic scrub scheduled @ 2024-03-10T16:53:07.157671+0000 17 0 2024-03-09T16:53:13.156 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-03-09T16:53:13.156 INFO:tasks.workunit.client.0.smithi146.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T16:53:13.157 INFO:tasks.workunit.client.0.smithi146.stderr:dumped pgs 2024-03-09T16:53:13.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-03-09T16:53:13.167 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T16:53:13.167 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr:1 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr:2' 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:13.437 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T16:53:13.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051590 2024-03-09T16:53:13.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051590 2024-03-09T16:53:13.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051590' 2024-03-09T16:53:13.540 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:13.540 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T16:53:13.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149703 2024-03-09T16:53:13.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149703 2024-03-09T16:53:13.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051590 1-111669149703' 2024-03-09T16:53:13.644 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T16:53:13.644 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051590 1-111669149703 2-64424509461' 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051590 2024-03-09T16:53:13.750 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:13.751 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T16:53:13.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051590 2024-03-09T16:53:13.751 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:13.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051590 2024-03-09T16:53:13.752 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2024-03-09T16:53:13.753 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.0 seq 124554051590 2024-03-09T16:53:13.753 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:53:14.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051590 2024-03-09T16:53:14.078 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T16:53:15.079 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T16:53:15.079 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T16:53:15.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051590 2024-03-09T16:53:15.360 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:15.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149703 2024-03-09T16:53:15.361 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:15.362 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T16:53:15.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149703 2024-03-09T16:53:15.363 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:15.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149703 2024-03-09T16:53:15.364 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149703' 2024-03-09T16:53:15.364 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.1 seq 111669149703 2024-03-09T16:53:15.364 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T16:53:15.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149703 2024-03-09T16:53:15.638 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T16:53:15.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-09T16:53:15.639 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T16:53:15.640 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T16:53:15.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-09T16:53:15.641 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T16:53:15.642 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-09T16:53:15.642 INFO:tasks.workunit.client.0.smithi146.stdout:waiting osd.2 seq 64424509461 2024-03-09T16:53:15.643 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-09T16:53:15.643 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T16:53:15.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-09T16:53:15.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-09T16:53:15.914 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes": 210, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects": 30, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_object_clones": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_object_copies": 90, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_missing_on_primary": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_missing": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_degraded": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_misplaced": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_unfound": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_dirty": 30, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_whiteouts": 0, 2024-03-09T16:53:16.021 INFO:tasks.workunit.client.0.smithi146.stdout: "num_read": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_read_kb": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_write": 30, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_write_kb": 30, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_scrub_errors": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shallow_scrub_errors": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_deep_scrub_errors": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_recovered": 26, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes_recovered": 182, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_keys_recovered": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_omap": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_hit_set_archive": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_bytes_hit_set_archive": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_kb": 0, 2024-03-09T16:53:16.022 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_kb": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_promote": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_mode_high": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_flush_mode_low": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_mode_some": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_evict_mode_full": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_pinned": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_legacy_snapsets": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_large_omap_objects": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_manifest": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_omap_bytes": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_omap_keys": 0, 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout: "num_objects_repaired": 26 2024-03-09T16:53:16.023 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:53:16.024 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-09T16:53:16.024 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T16:53:16.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-03-09T16:53:16.128 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-03-09T16:53:16.129 INFO:tasks.workunit.client.0.smithi146.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-03-09T16:53:16.130 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-03-09T16:53:16.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T16:53:16.131 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-03-09T16:53:16.131 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-03-09T16:53:16.132 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-03-09T16:53:16.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-09T16:53:16.133 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-09T16:53:16.388 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "osd": 0, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "up_from": 29, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "seq": 124554051590, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "num_pgs": 1, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "num_osds": 1, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_osds": 1, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T16:53:16.399 INFO:tasks.workunit.client.0.smithi146.stdout: "kb": 104857600, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used": 28472, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_data": 368, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_omap": 4, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_used_meta": 28091, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "kb_avail": 104829128, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "statfs": { 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "total": 107374182400, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "available": 107345027072, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "internally_reserved": 0, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "allocated": 376832, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "data_stored": 202971, 2024-03-09T16:53:16.400 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed": 0, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_allocated": 0, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "data_compressed_original": 0, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "omap_allocated": 4402, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "internal_metadata": 28765902 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "hb_peers": [ 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: 1, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: 2 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: ], 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "snap_trim_queue_len": 0, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "num_snap_trimming": 0, 2024-03-09T16:53:16.401 INFO:tasks.workunit.client.0.smithi146.stdout: "num_shards_repaired": 26, 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "op_queue_age_hist": { 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "histogram": [], 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "upper_bound": 1 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "perf_stat": { 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ms": 0, 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ms": 0, 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "commit_latency_ns": 0, 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: "apply_latency_ns": 0 2024-03-09T16:53:16.402 INFO:tasks.workunit.client.0.smithi146.stdout: }, 2024-03-09T16:53:16.403 INFO:tasks.workunit.client.0.smithi146.stdout: "alerts": [] 2024-03-09T16:53:16.403 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-03-09T16:53:16.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-09T16:53:16.403 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-09T16:53:16.653 INFO:tasks.workunit.client.0.smithi146.stderr:dumped all 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T16:53:16.667 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T16:53:16.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T16:53:16.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T16:53:16.668 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T16:53:16.775 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T16:53:16.776 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T16:53:16.776 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T16:53:16.777 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T16:53:16.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T16:53:16.778 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T16:53:16.778 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T16:53:16.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:53:16.779 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T16:53:16.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T16:53:16.780 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T16:53:16.780 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:53:16.782 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-09T16:53:16.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-09T16:53:16.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-09T16:53:16.783 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-03-09T16:53:16.784 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T16:53:16.793 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T16:53:16.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T16:53:16.794 INFO:tasks.workunit.client.0.smithi146.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69596 2024-03-09T16:53:16.794 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69596 2024-03-09T16:53:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T16:53:16.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T16:53:16.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T16:53:16.795 INFO:tasks.workunit.client.0.smithi146.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-09T16:53:16.796 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-03-09T16:53:16.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T16:53:17.114 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi146 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-09T16:53:17.402 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0059826eabf7419f90af87dd7f4156ed Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_a96b1c0427a378792255b04333efc5a5569fd416/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi146 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-09T16:53:17.406 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-09T16:53:17.419 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-09T16:53:17.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T16:53:17.477 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-09T16:53:17.487 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-09T16:53:17.487 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T16:53:17.506 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 15 +2411us[+2411us] +/- 98ms 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 8 377 81 -1071us[-1065us] +/- 78ms 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 79 -290us[ -290us] +/- 78ms 2024-03-09T16:53:17.511 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T16:53:17.511 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-09T16:53:17.520 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-09T16:53:17.520 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-09T16:53:17.529 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-09T16:53:17.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-09T16:53:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T16:53:17.645 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-09T16:53:17.654 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-09T16:53:17.665 INFO:teuthology.task.internal:Duration was 5579.032685 seconds 2024-03-09T16:53:17.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-09T16:53:17.675 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-09T16:53:17.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T16:53:17.728 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T16:53:18.121 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-09T16:53:18.121 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-09T16:53:18.121 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T16:53:18.180 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-09T16:53:18.181 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T16:53:18.251 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-09T16:53:18.251 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T16:53:18.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-09T16:53:18.397 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-09T16:53:18.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T16:53:18.424 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-09T16:53:18.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:53:18.491 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-09T16:53:18.516 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T16:53:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T16:53:18.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-09T16:53:18.581 INFO:teuthology.task.internal:Transferring archived files... 2024-03-09T16:53:18.582 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184/remote/smithi146 2024-03-09T16:53:18.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T16:53:19.716 INFO:teuthology.task.internal:Removing archive directory... 2024-03-09T16:53:19.717 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T16:53:19.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-09T16:53:19.770 INFO:teuthology.task.internal:Not uploading archives. 2024-03-09T16:53:19.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-09T16:53:19.780 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-09T16:53:19.780 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T16:53:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 16:53 /home/ubuntu/cephtest 2024-03-09T16:53:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: 403053 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 15:26 /home/ubuntu/cephtest/mnt.0 2024-03-09T16:53:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: 403054 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 15:37 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T16:53:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: 403095 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 15:37 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T16:53:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: 417241 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 16:53 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-09T16:53:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T16:53:19.816 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-09T16:53:19.816 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-09T16:53:19.821 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-09T16:53:19.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-09T16:53:19.844 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi146 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-09T16:53:19.844 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T16:53:19.869 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-09T16:53:19.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T16:53:19.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:19:51-rados-wip-yuri2-testing-2024-03-01-1606-distro-default-smithi/7587184', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 15:11:16.655937', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO829iPB/sWh84BWXixg0pjGsAKBKizpNi5TGVdn+npRBJu41t1RoVagiFjvfgVpmunoj/e8FiMAk3An6Io45Gg='} 2024-03-09T16:53:19.938 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-09T16:53:19.939 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T16:53:19.967 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-09T16:53:19.967 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T16:53:19.993 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-09T16:53:23.995 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T16:53:24.106 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-09T16:53:28.110 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T16:53:28.220 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-09T16:53:28.321 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-09T16:53:28.427 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 5579.032685279846 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi146 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a96b1c0427a378792255b04333efc5a5569fd416 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0059826eabf7419f90af87dd7f4156ed status: fail success: false 2024-03-09T16:53:28.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T16:53:28.523 INFO:teuthology.run:FAIL